[2024-04-24 10:00:46.901009] INFO New syslog file info: [address: "10.0.0.3:2882", observer version: OceanBase_CE 4.2.2.0, revision: 100010012024022719-c984fe7cb7a4cef85a40323a0d073f0c9b7b8235, sysname: Linux, os release: 4.18.0-193.28.1.el8_2.x86_64, machine: x86_64, tz GMT offset: 08:00] [2024-04-24 10:00:46.877154] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:192}, multi_version_start=1713895325234618001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.877694] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60294}, multi_version_start=1713895325234618001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.877806] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1457) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2B9DC6169-0-0] [lt=106] receive register_req from acceptor, ignore(self="10.0.0.3:2882", palf_id=1001, child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924046873287, keepalive_ts:-1}) [2024-04-24 10:00:46.877935] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1476) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2B9DC6169-0-0] [lt=119] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="10.0.0.3:2882", child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924046873287, keepalive_ts:-1}, is_to_leader=true) [2024-04-24 10:00:46.878190] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60214}, multi_version_start=1713895325234618001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.878679] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60398}, multi_version_start=1713895325234618001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.879172] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101072}, multi_version_start=1713895325734725000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.879585] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50431}, multi_version_start=1713895325734725000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.880092] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=131] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101088}, multi_version_start=1713895325734725000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.880513] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=53] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50150}, multi_version_start=1713895325734725000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.881055] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101069}, multi_version_start=1713895326109843000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.881515] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:244}, multi_version_start=1713895326109843001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.881981] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50245}, multi_version_start=1713895326432848000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.882378] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60493}, multi_version_start=1713895326432848002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.882854] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60106}, multi_version_start=1713895326432848002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.883327] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=115] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:191}, multi_version_start=1713895326432848002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.883832] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101060}, multi_version_start=1713895326606094001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.884267] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=107] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:252}, multi_version_start=1713895326738380000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.884687] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60336}, multi_version_start=1713895326738380000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.885197] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50442}, multi_version_start=1713895326738380000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.885657] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101022}, multi_version_start=1713895326738380000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.886111] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [874531][T1004_TabletShe][T1004][Y0-0000000000000000-0-0] [lt=110] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-04-24 10:00:46.886170] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:151}, multi_version_start=1713895326738380003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.886214] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [874531][T1004_TabletShe][T1004][Y0-0000000000000000-0-0] [lt=72] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=3, times=9343) [2024-04-24 10:00:46.886594] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=52] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50110}, multi_version_start=1713895326738380003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.887112] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50252}, multi_version_start=1713895326951164002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.887566] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50471}, multi_version_start=1713895326951164002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.888076] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:325}, multi_version_start=1713895326951164002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.888499] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50214}, multi_version_start=1713895327141625002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.888924] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=54] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:305}, multi_version_start=1713895327141625002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.889361] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50428}, multi_version_start=1713895327330595002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.889793] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50113}, multi_version_start=1713895327330595002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.890227] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50441}, multi_version_start=1713895327330595003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.890577] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=52] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101035}, multi_version_start=1713895327330595003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.891066] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:5}, multi_version_start=1713895203755260001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.891560] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60327}, multi_version_start=1713895327330595003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.892003] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50145}, multi_version_start=1713895327330595003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.892247] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=63] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:46.892371] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=88] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:46.892437] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=59] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:46.892462] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=109] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60426}, multi_version_start=1713895327552505005, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.893033] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101058}, multi_version_start=1713895327552505006, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.893522] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:407}, multi_version_start=1713895327552505006, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.893988] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101081}, multi_version_start=1713895327552505006, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.894152] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [862131][T1002_Occam][T1002][YB420A000003-000616C2AFD988F0-0-0] [lt=80] for each ls(succ_num=3, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:46.894467] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=51] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:245}, multi_version_start=1713895327552505006, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.894991] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101086}, multi_version_start=1713895327741201003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.895237] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [861461][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=100] skip inspect bad block(last_check_time=1713875167993818, last_macro_idx=-1) [2024-04-24 10:00:46.895524] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101052}, multi_version_start=1713895327741201003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.896011] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50133}, multi_version_start=1713895328038618000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.896516] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101019}, multi_version_start=1713895328038618001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.897027] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60157}, multi_version_start=1713895328038618001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.897489] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101079}, multi_version_start=1713895328038618001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.897974] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50307}, multi_version_start=1713895328242846001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.898356] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=54] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50272}, multi_version_start=1713895328242846001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.898860] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=52] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:238}, multi_version_start=1713895328242846001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.899363] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60306}, multi_version_start=1713895328440651001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.899929] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50333}, multi_version_start=1713895328440651001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.900314] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:150}, multi_version_start=1713895328440651001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.900824] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50238}, multi_version_start=1713895328744631000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.901355] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=147] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:406}, multi_version_start=1713895328744631000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.901980] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1713920414963170000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.902420] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60330}, multi_version_start=1713895328864188000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.902984] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50319}, multi_version_start=1713895328864188000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.903427] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50377}, multi_version_start=1713895329029695000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.903552] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=136] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8903270400, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:46.904058] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:396}, multi_version_start=1713895329029695000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.904609] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60407}, multi_version_start=1713895329233681002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.905216] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50104}, multi_version_start=1713895329233681002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.905746] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50425}, multi_version_start=1713895329233681002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.906375] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50436}, multi_version_start=1713895329468526001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.906859] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:474}, multi_version_start=1713895329563254000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.907455] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=179] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60148}, multi_version_start=1713895329563254000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.908232] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=131] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60004}, multi_version_start=1713895329563254000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.908812] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:426}, multi_version_start=1713895329563254004, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.909317] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50438}, multi_version_start=1713895329746301001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.909379] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=140] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-24 10:00:46.909607] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=116] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1713924046909508], gts:1713924046908812000, latest_srr:[mts=1713924046909508]}) [2024-04-24 10:00:46.909862] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=144] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1713924046909773], gts:1713924046908812000, latest_srr:[mts=1713924046909773]}) [2024-04-24 10:00:46.909941] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:235}, multi_version_start=1713895329746301001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.910088] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=121] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=587, get_gts_with_stc_cnt=51, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=46, try_wait_gts_elapse_cnt=0) [2024-04-24 10:00:46.910460] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50336}, multi_version_start=1713895329746301001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.910969] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:397}, multi_version_start=1713895329884992001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.911439] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:110}, multi_version_start=1713895329998116001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.912021] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60254}, multi_version_start=1713895329998116001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.912493] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:202}, multi_version_start=1713895329998116001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.913143] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60146}, multi_version_start=1713895329998116001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.913653] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:319}, multi_version_start=1713895330460296002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.914225] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50423}, multi_version_start=1713895330460296002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.914709] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60111}, multi_version_start=1713895330827333000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.915143] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50208}, multi_version_start=1713895330827333000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.915609] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50223}, multi_version_start=1713895330943452001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.916145] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=98] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60244}, multi_version_start=1713895330943452001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.916710] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60439}, multi_version_start=1713895330943452001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.917300] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60003}, multi_version_start=1713895330943452001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.917965] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:253}, multi_version_start=1713895330943452001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.918491] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:382}, multi_version_start=1713895331165954000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.919015] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50157}, multi_version_start=1713895331943969002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.919451] INFO eloop_run (eloop.c:143) [861404][pnio2][T0][Y0-0000000000000000-0-0] [lt=61] PNIO [ratelimit] time: 1713924046919442, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007694, add_bytes: 0 [2024-04-24 10:00:46.919498] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1713895331943969002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.920034] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60149}, multi_version_start=1713895331943969003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.920440] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50189}, multi_version_start=1713895332174763001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.920906] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:291}, multi_version_start=1713895332174763001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.921303] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60471}, multi_version_start=1713895332174763001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.921738] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1713895332174763001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.922277] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60332}, multi_version_start=1713895332327238002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.922722] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60405}, multi_version_start=1713895332327238002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.923238] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60150}, multi_version_start=1713895332327238002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.923789] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60213}, multi_version_start=1713895332591727000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.924189] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=74][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:46.924397] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=190] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60375}, multi_version_start=1713895332591727000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.925013] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50188}, multi_version_start=1713895332591727002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.925365] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=86] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:46.925504] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101043}, multi_version_start=1713895332591727002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.926105] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50378}, multi_version_start=1713895332591727002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.926646] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50225}, multi_version_start=1713895332591727005, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.927202] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:167}, multi_version_start=1713895332591727005, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.927696] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60121}, multi_version_start=1713895332753763002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.928263] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:149}, multi_version_start=1713895332753763002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.928752] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60151}, multi_version_start=1713895332753763002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.929238] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50232}, multi_version_start=1713895332870922003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.929764] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50236}, multi_version_start=1713895332870922004, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.930327] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:143}, multi_version_start=1713895332870922004, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.930903] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:189}, multi_version_start=1713895332870922004, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.931403] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60246}, multi_version_start=1713895333034098002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.931918] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101032}, multi_version_start=1713895333034098003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.932319] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:208}, multi_version_start=1713895333034098003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.932770] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50356}, multi_version_start=1713895333034098003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.933186] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:326}, multi_version_start=1713895333287624002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.933669] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50331}, multi_version_start=1713895333287624002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.934115] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=54] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50332}, multi_version_start=1713895333287624003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.934672] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50474}, multi_version_start=1713895333287624003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.935266] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50248}, multi_version_start=1713895333287624003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.935772] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:107}, multi_version_start=1713895333575598001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.936266] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50324}, multi_version_start=1713895333575598001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.936821] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60168}, multi_version_start=1713895333575598003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.937337] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=137] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:102}, multi_version_start=1713895333575598003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.937911] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60236}, multi_version_start=1713895333873600000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.938321] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50294}, multi_version_start=1713895333873600000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.938902] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101028}, multi_version_start=1713895333927910000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.939351] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:239}, multi_version_start=1713895334281133001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.939790] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50253}, multi_version_start=1713895334463311000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.940351] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=197] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50132}, multi_version_start=1713895334463311000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.940461] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=113] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:46.940778] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60378}, multi_version_start=1713895334463311000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.941275] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101021}, multi_version_start=1713895334588928001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.942048] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=238] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50343}, multi_version_start=1713895334588928001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.942380] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [862266][T1002_LSArchive][T1002][YB420A000003-000616C2AE58F0E7-0-0] [lt=88] gc stale ls task succ [2024-04-24 10:00:46.942314] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2627-0-0] [lt=82][errcode=-4023] tx is 2pc logging(this={this:0x14b0fbedf050, ref:3, trans_id:{txid:338711}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924076767546, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046792910], ctx_create_time:1713924046778243}{ls_id:{id:1}, session_id:3221795357, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046789276, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046791807000, v:0}, log_ts:{val:1713924046791807000, v:0}, lsn:{lsn:298513878}, submit_ts:1713924046792973}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:338711}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338711}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046791807000, v:0}, start_scn:{val:1713924046791807000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924046791807000, v:0} commit_version={val:1713924046791807000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338711} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=215183838 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046791807000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046792973, block_frozen_memtable:null}) [2024-04-24 10:00:46.942433] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [862460][T1001_L0_G0][T1001][YB420A000002-000616C2B19C24D9-0-0] [lt=99] current clock is (clock_=0, max_seq_=0, mem_limit=192099116, cur_mem_hold=130023424, attr_.tenant_id_=1001) [2024-04-24 10:00:46.942537] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60324}, multi_version_start=1713895334588928001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.942487] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2627-0-0] [lt=164][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fbedf050, ref:3, trans_id:{txid:338711}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924076767546, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046792910], ctx_create_time:1713924046778243}{ls_id:{id:1}, session_id:3221795357, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046789276, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046791807000, v:0}, log_ts:{val:1713924046791807000, v:0}, lsn:{lsn:298513878}, submit_ts:1713924046792973}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:338711}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338711}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046791807000, v:0}, start_scn:{val:1713924046791807000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924046791807000, v:0} commit_version={val:1713924046791807000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338711} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=215183838 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046791807000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046792973, block_frozen_memtable:null}) [2024-04-24 10:00:46.942645] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2627-0-0] [lt=151][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338711}) [2024-04-24 10:00:46.942732] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2627-0-0] [lt=75][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338711}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924046937454, cluster_id:1713874979}, expire_ts:1713924076788074, commit_start_scn:{val:1713924046780604000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:46.942814] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2627-0-0] [lt=68][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338711}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924046937454, cluster_id:1713874979}, expire_ts:1713924076788074, commit_start_scn:{val:1713924046780604000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:46.943126] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50250}, multi_version_start=1713895334765448001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.943101] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A0-0-0] [lt=65][errcode=-4023] tx is 2pc logging(this={this:0x14b148003290, ref:4, trans_id:{txid:176905}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924076581982, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589598], ctx_create_time:1713924046582067}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046580909, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452000, v:0}, log_ts:{val:1713924046589452000, v:0}, lsn:{lsn:79580022}, submit_ts:1713924046589452}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176905}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176905}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452000, v:0}, start_scn:{val:1713924046589452000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465665976 max_table_version=1713877465684248 trans_version={val:1713924046589452000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176905} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=924633448 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:46.943251] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A0-0-0] [lt=144][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b148003290, ref:4, trans_id:{txid:176905}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924076581982, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589598], ctx_create_time:1713924046582067}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046580909, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452000, v:0}, log_ts:{val:1713924046589452000, v:0}, lsn:{lsn:79580022}, submit_ts:1713924046589452}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176905}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176905}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452000, v:0}, start_scn:{val:1713924046589452000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465665976 max_table_version=1713877465684248 trans_version={val:1713924046589452000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176905} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=924633448 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:46.943385] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A0-0-0] [lt=118][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176905}) [2024-04-24 10:00:46.943465] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A0-0-0] [lt=74][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176905}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924046938561, cluster_id:1713874979}, expire_ts:1713924076505566, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:46.943465] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2325174][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A1-0-0] [lt=75][errcode=-4023] tx is 2pc logging(this={this:0x14b055e1a050, ref:3, trans_id:{txid:176904}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138442, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589701], ctx_create_time:1713924046585672}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046585674, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452001, v:0}, log_ts:{val:1713924046589452001, v:0}, lsn:{lsn:79580890}, submit_ts:1713924046589452}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176904}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176904}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452001, v:0}, start_scn:{val:1713924046589452001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877457407528 trans_version={val:1713924046589452001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176904} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3140423830 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:46.943561] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60292}, multi_version_start=1713895334878066000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.943554] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A0-0-0] [lt=75][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176905}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924046938561, cluster_id:1713874979}, expire_ts:1713924076505566, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:46.943662] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2325174][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A1-0-0] [lt=183][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e1a050, ref:3, trans_id:{txid:176904}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138442, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589701], ctx_create_time:1713924046585672}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046585674, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452001, v:0}, log_ts:{val:1713924046589452001, v:0}, lsn:{lsn:79580890}, submit_ts:1713924046589452}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176904}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176904}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452001, v:0}, start_scn:{val:1713924046589452001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877457407528 trans_version={val:1713924046589452001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176904} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3140423830 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:46.943857] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2325174][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A1-0-0] [lt=182][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176904}) [2024-04-24 10:00:46.943965] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2325174][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A1-0-0] [lt=102][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176904}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924046938875, cluster_id:1713874979}, expire_ts:1713925045138403, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:46.944095] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2325174][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A1-0-0] [lt=113][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176904}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924046938875, cluster_id:1713874979}, expire_ts:1713925045138403, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:46.944165] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60196}, multi_version_start=1713895334878066000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.944597] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60239}, multi_version_start=1713895335575481000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.945071] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50498}, multi_version_start=1713895335575481000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.945517] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:111}, multi_version_start=1713895335575481003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.946019] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50111}, multi_version_start=1713895335575481003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.946569] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:209}, multi_version_start=1713895335832203000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.947187] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=130] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60369}, multi_version_start=1713895335832203000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.947832] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60110}, multi_version_start=1713895336127794000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.948511] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=156] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:443}, multi_version_start=1713895336127794000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.948672] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E93-0-0] [lt=71][errcode=-4023] tx is 2pc logging(this={this:0x14b168f2d710, ref:3, trans_id:{txid:877456}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924646597291, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046606334], ctx_create_time:1713924046597291}{ls_id:{id:1001}, session_id:3221852891, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046603230, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046605790000, v:0}, log_ts:{val:1713924046605790000, v:0}, lsn:{lsn:1002185585}, submit_ts:1713924046605790}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877456}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877456}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046605790000, v:0}, start_scn:{val:1713924046605790000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924046605790000, v:0} commit_version={val:1713924046605790000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877456} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=737308951 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046605790000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046605790, block_frozen_memtable:null}) [2024-04-24 10:00:46.948860] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E93-0-0] [lt=179][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b168f2d710, ref:3, trans_id:{txid:877456}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924646597291, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046606334], ctx_create_time:1713924046597291}{ls_id:{id:1001}, session_id:3221852891, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046603230, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046605790000, v:0}, log_ts:{val:1713924046605790000, v:0}, lsn:{lsn:1002185585}, submit_ts:1713924046605790}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877456}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877456}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046605790000, v:0}, start_scn:{val:1713924046605790000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924046605790000, v:0} commit_version={val:1713924046605790000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877456} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=737308951 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046605790000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046605790, block_frozen_memtable:null}) [2024-04-24 10:00:46.949015] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50375}, multi_version_start=1713895336772307000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.949043] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E93-0-0] [lt=176][errcode=-4023] commit fail(ret=-4023, coord={id:1001}, tx_id={txid:877456}) [2024-04-24 10:00:46.949133] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E93-0-0] [lt=83][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877456}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924046944168, cluster_id:1713874979}, expire_ts:1713924558320000, commit_start_scn:{val:1713924046594982000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:46.949224] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E93-0-0] [lt=78][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877456}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924046944168, cluster_id:1713874979}, expire_ts:1713924558320000, commit_start_scn:{val:1713924046594982000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:46.949561] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101037}, multi_version_start=1713895336772307000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.950155] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=125] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:263}, multi_version_start=1713895336772307000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.950724] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:135}, multi_version_start=1713895336866013000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.951434] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=53] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:335}, multi_version_start=1713895336866013001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.952133] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=124] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101054}, multi_version_start=1713895336866013001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.952686] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=104] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:153}, multi_version_start=1713895337199213000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.953199] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50165}, multi_version_start=1713895337199213000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.953589] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60433}, multi_version_start=1713895337199213002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.954174] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=192] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50005}, multi_version_start=1713895337199213002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.954729] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101041}, multi_version_start=1713895337390526001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.954969] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=111][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:46.955249] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50369}, multi_version_start=1713895337390526001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.955699] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60210}, multi_version_start=1713895337390526001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.956222] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101024}, multi_version_start=1713895337390526001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.956750] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50147}, multi_version_start=1713895337496921003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.957300] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:236}, multi_version_start=1713895337496921004, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.957845] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50316}, multi_version_start=1713895337691212001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.958355] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=130] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:237}, multi_version_start=1713895337691212001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.958776] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60225}, multi_version_start=1713895337691212002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.959257] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60131}, multi_version_start=1713895337691212002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.959699] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50190}, multi_version_start=1713895337862739001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.960235] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60423}, multi_version_start=1713895338004950001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.960833] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50439}, multi_version_start=1713895338004950001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.961357] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101055}, multi_version_start=1713895338004950002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.961908] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:113}, multi_version_start=1713895338138173001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.962368] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50290}, multi_version_start=1713895338138173001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.962861] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60437}, multi_version_start=1713895338138173001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.963468] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=149] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50433}, multi_version_start=1713895338427806000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.963737] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [862399][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=143] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2024-04-24 10:00:46.964047] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50135}, multi_version_start=1713895338427806000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.964414] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=77] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:46.964475] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:471}, multi_version_start=1713895338427806000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.964501] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=81] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:46.964932] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:450}, multi_version_start=1713895338427806000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.965437] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1713895338613855000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.965954] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60397}, multi_version_start=1713895338613855000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.966493] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:375}, multi_version_start=1713895338613855000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.967022] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:3}, multi_version_start=1713895338613855002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.967516] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60112}, multi_version_start=1713895338613855002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.968030] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60333}, multi_version_start=1713895341218132000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.968483] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:168}, multi_version_start=1713895341218132000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.969026] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101040}, multi_version_start=1713895342992984000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.969519] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:259}, multi_version_start=1713895342992984000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.970261] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=188] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50491}, multi_version_start=1713895342992984000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.970837] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:248}, multi_version_start=1713895343096341001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.971370] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=157] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60153}, multi_version_start=1713895343096341001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.971991] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101047}, multi_version_start=1713895343096341001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.972443] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:106}, multi_version_start=1713895343377566000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.973053] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60307}, multi_version_start=1713895343377566000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.973547] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101059}, multi_version_start=1713895343377566000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.974135] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=131] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50131}, multi_version_start=1713895343377566000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.974537] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50262}, multi_version_start=1713895343542771002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.975051] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=109] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:324}, multi_version_start=1713895343542771003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.975465] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=51] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:441}, multi_version_start=1713895343542771003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.976054] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:130}, multi_version_start=1713895343698999001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.976506] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:262}, multi_version_start=1713895343698999001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.977142] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50143}, multi_version_start=1713895343931695001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.977562] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:114}, multi_version_start=1713895343931695001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.978185] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60240}, multi_version_start=1713895343931695001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.978697] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:206}, multi_version_start=1713895344148093000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.979273] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:207}, multi_version_start=1713895344148093000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.979808] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:498}, multi_version_start=1713895344148093000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.980333] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=151] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:380}, multi_version_start=1713895344554951001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.980728] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1713895344554951001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.981300] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60413}, multi_version_start=1713895344554951001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.981790] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50166}, multi_version_start=1713895344554951001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.982356] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=142] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60356}, multi_version_start=1713895344554951001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.982832] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50434}, multi_version_start=1713895344796721002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.983387] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:295}, multi_version_start=1713895344796721002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.984019] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50180}, multi_version_start=1713895344796721002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.984501] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60343}, multi_version_start=1713895345423403000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.985012] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60143}, multi_version_start=1713895346301396001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.985438] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60406}, multi_version_start=1713895346301396001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.985948] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60255}, multi_version_start=1713895346301396001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.986376] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=53] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50242}, multi_version_start=1713895346301396001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.986944] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:436}, multi_version_start=1713895346517967003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.987471] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101046}, multi_version_start=1713895346698989000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.988011] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60427}, multi_version_start=1713895346698989000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.988484] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60105}, multi_version_start=1713895346698989000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.989044] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60252}, multi_version_start=1713895346950682002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.988999] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2606-0-0] [lt=107][errcode=-4023] tx is 2pc logging(this={this:0x14b15a16b090, ref:3, trans_id:{txid:391371}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924076773625, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046800195], ctx_create_time:1713924046784446}{ls_id:{id:1}, session_id:3221795340, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046791808, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046799990000, v:0}, log_ts:{val:1713924046799990000, v:0}, lsn:{lsn:314127056}, submit_ts:1713924046799990}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:391371}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:391371}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046799990000, v:0}, start_scn:{val:1713924046799990000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875442714112 max_table_version=1713875442714112 trans_version={val:1713924046799990000, v:0} commit_version={val:1713924046799990000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:391371} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=2612980561 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046799990000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046799990, block_frozen_memtable:null}) [2024-04-24 10:00:46.989243] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2606-0-0] [lt=236][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b15a16b090, ref:3, trans_id:{txid:391371}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924076773625, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046800195], ctx_create_time:1713924046784446}{ls_id:{id:1}, session_id:3221795340, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046791808, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046799990000, v:0}, log_ts:{val:1713924046799990000, v:0}, lsn:{lsn:314127056}, submit_ts:1713924046799990}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:391371}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:391371}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046799990000, v:0}, start_scn:{val:1713924046799990000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875442714112 max_table_version=1713875442714112 trans_version={val:1713924046799990000, v:0} commit_version={val:1713924046799990000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:391371} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=2612980561 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046799990000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046799990, block_frozen_memtable:null}) [2024-04-24 10:00:46.989356] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2606-0-0] [lt=105][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:391371}) [2024-04-24 10:00:46.989422] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2606-0-0] [lt=56][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1001, tx_id:{txid:391371}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924046984433, cluster_id:1713874979}, expire_ts:1713924076795547, commit_start_scn:{val:1713924046786976000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:46.989528] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2606-0-0] [lt=76][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1001, tx_id:{txid:391371}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924046984433, cluster_id:1713874979}, expire_ts:1713924076795547, commit_start_scn:{val:1713924046786976000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:46.989569] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:272}, multi_version_start=1713895347191740000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.990132] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=109] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50295}, multi_version_start=1713895347191740000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.990630] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50382}, multi_version_start=1713895347191740000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.991146] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50394}, multi_version_start=1713895347440115001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.991639] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50152}, multi_version_start=1713895347440115001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.992179] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:251}, multi_version_start=1713895347663452001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.992479] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=66] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:46.992648] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=124] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:46.992712] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50490}, multi_version_start=1713895347663452001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.992760] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=100] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:46.993257] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60191}, multi_version_start=1713895347978073000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.993344] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [862269][T1002_ArcSender][T1002][YB420A000003-000616C2AE88F0E7-0-0] [lt=232][errcode=-4018] try free send task failed(ret=-4018) [2024-04-24 10:00:46.993655] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2720) [861318][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4002] get invalid Ethernet speed, use default(devname="eth0") [2024-04-24 10:00:46.993739] WDIAG [SERVER] runTimerTask (ob_server.cpp:3239) [861318][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4002] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4002, ret="OB_INVALID_ARGUMENT") [2024-04-24 10:00:46.993787] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:223}, multi_version_start=1713895347978073000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.994276] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=147] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50291}, multi_version_start=1713895347978073000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.994998] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=53] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101045}, multi_version_start=1713895347978073000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.995463] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60331}, multi_version_start=1713895348151265000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.996163] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=129] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:131}, multi_version_start=1713895348151265000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.996731] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60442}, multi_version_start=1713895348151265002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.997223] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60284}, multi_version_start=1713895348151265003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.997722] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50243}, multi_version_start=1713895348448723000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.998255] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60316}, multi_version_start=1713895348448723000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.998744] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60182}, multi_version_start=1713895348448723001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.999033] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [862447][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=96] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:46.999247] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:246}, multi_version_start=1713895348448723001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:46.999824] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60499}, multi_version_start=1713895348448723001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.000294] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=121] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101078}, multi_version_start=1713895348950511001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.000802] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:148}, multi_version_start=1713895348950511001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.001289] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=106] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:408}, multi_version_start=1713895348950511001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.001781] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101049}, multi_version_start=1713895349062850003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.002235] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:247}, multi_version_start=1713895349062850003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.002676] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:220}, multi_version_start=1713895349232620000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.003207] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50151}, multi_version_start=1713895349232620000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.003687] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50209}, multi_version_start=1713895349869660002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.004270] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:221}, multi_version_start=1713895349869660002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.004486] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:488) [861547][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=187] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2024-04-24 10:00:47.004773] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50255}, multi_version_start=1713895350455723000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.005291] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60135}, multi_version_start=1713895350455723001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.005804] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:169}, multi_version_start=1713895350455723001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.006339] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=152] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50202}, multi_version_start=1713895350455723003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.006821] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:213}, multi_version_start=1713895350455723003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.007368] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=155] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50330}, multi_version_start=1713895350455723003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.007945] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101091}, multi_version_start=1713895350867694001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.008393] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50426}, multi_version_start=1713895350867694001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.008999] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60220}, multi_version_start=1713895350867694001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.009239] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:503) [2324801][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=116] worker thread exit(tenant_->id()=1001, workers_.get_size()=1) [2024-04-24 10:00:47.009370] INFO [SHARE] end_run (ob_tenant_base.cpp:332) [2324801][T1001_L0_G1][T0][Y0-0000000000000000-0-0] [lt=117] tenant thread end_run(id_=1001, ret=0, thread_count_=165) [2024-04-24 10:00:47.009466] INFO unregister_pm (ob_page_manager.cpp:50) [2324801][T1001_L0_G1][T0][Y0-0000000000000000-0-0] [lt=77] unregister pm finish(&pm=0x14b10e8d4d40, pm.get_tid()=2324801) [2024-04-24 10:00:47.009517] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=106] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:356}, multi_version_start=1713895351068755000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.010114] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101039}, multi_version_start=1713895351068755000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.010579] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101029}, multi_version_start=1713895351068755000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.011171] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:283}, multi_version_start=1713895353276373000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.011642] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60408}, multi_version_start=1713895353276373000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.012150] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60235}, multi_version_start=1713895353276373000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.012556] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60215}, multi_version_start=1713895353276373000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.013043] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60325}, multi_version_start=1713895353276373000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.013568] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50397}, multi_version_start=1713895357700275000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.014137] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:433}, multi_version_start=1713895359849428002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.014610] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60261}, multi_version_start=1713895360039120000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.015111] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60206}, multi_version_start=1713895360039120000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.015593] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:333}, multi_version_start=1713895360404030001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.016146] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50308}, multi_version_start=1713895360404030001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.016741] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=107] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50206}, multi_version_start=1713895360564628001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.017202] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50413}, multi_version_start=1713895360564628001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.017824] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=126] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:424}, multi_version_start=1713895360564628003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.018390] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=131] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:100004}, multi_version_start=1713895360564628003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.018981] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60250}, multi_version_start=1713895360792197001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.019589] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101070}, multi_version_start=1713895360792197002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.020303] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=149] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1713895360792197002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.021059] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:435}, multi_version_start=1713895360792197002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.021603] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=104] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:254}, multi_version_start=1713895361207089001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.022145] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101075}, multi_version_start=1713895361473211001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.022680] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50182}, multi_version_start=1713895361856415000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.023237] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60373}, multi_version_start=1713895361856415000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.023927] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60197}, multi_version_start=1713895361856415002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.024336] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=85][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.024485] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:181}, multi_version_start=1713895361856415002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.025104] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101077}, multi_version_start=1713895362394200000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.025701] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1713895362811188000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.026297] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:142}, multi_version_start=1713895363304234001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.026940] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=107] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101071}, multi_version_start=1713895363702176000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.027448] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:190}, multi_version_start=1713895363702176000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.028082] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101012}, multi_version_start=1713895363803325000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.028600] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50220}, multi_version_start=1713895363803325001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.029242] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101065}, multi_version_start=1713895363963362000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.029756] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:438}, multi_version_start=1713895363963362000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.030316] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=162] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:214}, multi_version_start=1713895363963362000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.030909] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:394}, multi_version_start=1713895203755260001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.031127] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=136] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-04-24 10:00:47.031371] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:329}, multi_version_start=1713895364362626000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.031946] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:100003}, multi_version_start=1713895364722141000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.032392] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60435}, multi_version_start=1713895364722141000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.033005] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:105}, multi_version_start=1713895364722141000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.033538] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1713920414963170000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.034087] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50435}, multi_version_start=1713895364908553000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.034565] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101014}, multi_version_start=1713895364908553000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.035123] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60308}, multi_version_start=1713895365105089000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.035542] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1713895365105089001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.036063] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50443}, multi_version_start=1713895365105089001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.036584] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=120] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:261}, multi_version_start=1713895365190426001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.037101] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:147}, multi_version_start=1713895365616741001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.037555] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=111] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50103}, multi_version_start=1713895365616741001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.038110] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101004}, multi_version_start=1713895365616741001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.038654] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60394}, multi_version_start=1713895365867080000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.039180] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:120}, multi_version_start=1713895365867080002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.039686] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60319}, multi_version_start=1713895365867080002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.040288] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60326}, multi_version_start=1713895365867080002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.040665] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=67] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.041006] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=148] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:379}, multi_version_start=1713895367825464000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.041600] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50326}, multi_version_start=1713895367932671000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.042081] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50221}, multi_version_start=1713895367932671000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.042689] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60283}, multi_version_start=1713895367932671000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.043203] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50396}, multi_version_start=1713895368036292001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.043779] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:413}, multi_version_start=1713895368036292001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.044298] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=144] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50235}, multi_version_start=1713895368036292001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.044233] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [874398][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=181] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}]) [2024-04-24 10:00:47.044794] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:240}, multi_version_start=1713895368233913001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.045313] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=122] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50213}, multi_version_start=1713895368233913002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.045852] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:133}, multi_version_start=1713895368233913002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.046337] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=121] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101050}, multi_version_start=1713895368416760001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.046855] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50407}, multi_version_start=1713895368416760001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.047424] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=133] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:100001}, multi_version_start=1713895368416760003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.048116] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [861771][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=160] current monitor number(seq_=-1) [2024-04-24 10:00:47.048224] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50284}, multi_version_start=1713895368416760003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.048774] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:243}, multi_version_start=1713895368416760003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.049461] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=139] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60498}, multi_version_start=1713895368416760003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.050170] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50327}, multi_version_start=1713895368723364000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.050675] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60436}, multi_version_start=1713895368723364000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.051310] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60223}, multi_version_start=1713895368723364000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.051838] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101056}, multi_version_start=1713895368723364000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.052414] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=132] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50210}, multi_version_start=1713895368832963002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.052940] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:434}, multi_version_start=1713895368832963002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.053223] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A2-0-0] [lt=130][errcode=-4023] tx is 2pc logging(this={this:0x14b147fffbd0, ref:4, trans_id:{txid:176906}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925046666492, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046708409], ctx_create_time:1713924046701144}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79581303}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046705015, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046707343000, v:0}, log_ts:{val:1713924046707343000, v:0}, lsn:{lsn:79582268}, submit_ts:1713924046708648}, this:0x14b148002b40, is_inited_:true, trans_id:{txid:176906}, ls_id:{id:1}, ctx:0x14b147fffbd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176906}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046707343000, v:0}, start_scn:{val:1713924046589452002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877459227536 max_table_version=1713877465782000 trans_version={val:1713924046707343000, v:0} commit_version={val:1713924046707343000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176906} ls_id=1 callback_alloc_count=5 callback_free_count=0 checksum=3165083175 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=8 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=8 unsynced_cnt=5 unsubmitted_cnt_=0 cb_statistics:[main=8, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046708648, block_frozen_memtable:null}) [2024-04-24 10:00:47.053442] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50244}, multi_version_start=1713895368832963003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.053425] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A2-0-0] [lt=192][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b147fffbd0, ref:4, trans_id:{txid:176906}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925046666492, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046708409], ctx_create_time:1713924046701144}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79581303}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046705015, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046707343000, v:0}, log_ts:{val:1713924046707343000, v:0}, lsn:{lsn:79582268}, submit_ts:1713924046708648}, this:0x14b148002b40, is_inited_:true, trans_id:{txid:176906}, ls_id:{id:1}, ctx:0x14b147fffbd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176906}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046707343000, v:0}, start_scn:{val:1713924046589452002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877459227536 max_table_version=1713877465782000 trans_version={val:1713924046707343000, v:0} commit_version={val:1713924046707343000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176906} ls_id=1 callback_alloc_count=5 callback_free_count=0 checksum=3165083175 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=8 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=8 unsynced_cnt=5 unsubmitted_cnt_=0 cb_statistics:[main=8, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046708648, block_frozen_memtable:null}) [2024-04-24 10:00:47.053677] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A2-0-0] [lt=243][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176906}) [2024-04-24 10:00:47.053750] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A2-0-0] [lt=67][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176906}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047038344, cluster_id:1713874979}, expire_ts:1713925046666035, commit_start_scn:{val:1713924046703599000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.053904] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A2-0-0] [lt=109][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176906}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047038344, cluster_id:1713874979}, expire_ts:1713925046666035, commit_start_scn:{val:1713924046703599000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.053980] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:180}, multi_version_start=1713895369006798001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.054532] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:490}, multi_version_start=1713895369122921000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.054997] INFO [STORAGE] exec_wait (ob_tmp_file_cache.cpp:1531) [1556243][T1004_ComTimerT][T1004][Y0-0000000000000000-0-0] [lt=129] succeed to do one round of tmp block io(ret=-4023, loop_nums=0, wait_io_cnt=0, washing_count=0, block_cache_num=0, page_cache_num=0) [2024-04-24 10:00:47.055113] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=76] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"10.0.0.3:2882", switch_to_leader_ts:-1}) [2024-04-24 10:00:47.055127] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60207}, multi_version_start=1713895369122921000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.055262] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=128] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"10.0.0.3:2882", switch_to_leader_ts:-1}) [2024-04-24 10:00:47.055358] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=89][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.055726] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=135] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101023}, multi_version_start=1713895369415713000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.056197] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:146}, multi_version_start=1713895369415713000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.056645] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101007}, multi_version_start=1713895369415713000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.057121] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101048}, multi_version_start=1713895369852368002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.057550] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1713895369852368003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.058087] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101026}, multi_version_start=1713895369852368003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.058535] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50260}, multi_version_start=1713895370012609001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.058821] INFO eloop_run (eloop.c:143) [861399][pnio1][T0][Y0-0000000000000000-0-0] [lt=101] PNIO [ratelimit] time: 1713924047058814, bytes: 14675583848, bw: 0.295428 MB/s, add_ts: 1003282, add_bytes: 310795 [2024-04-24 10:00:47.059070] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60189}, multi_version_start=1713895370117502000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.059517] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50211}, multi_version_start=1713895370117502000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.060079] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50215}, multi_version_start=1713895370117502000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.060501] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60335}, multi_version_start=1713895370224437002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.061010] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:428}, multi_version_start=1713895370224437003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.061472] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50106}, multi_version_start=1713895370224437003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.062078] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101025}, multi_version_start=1713895370224437003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.062517] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60188}, multi_version_start=1713895370540119000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.063256] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50196}, multi_version_start=1713895370540119000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.063816] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60132}, multi_version_start=1713895370540119000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.064338] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:104}, multi_version_start=1713895370540119003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.064615] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=56] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.064717] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=92] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.064940] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:4}, multi_version_start=1713895370540119003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.065450] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:100005}, multi_version_start=1713895370540119003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.066019] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:355}, multi_version_start=1713895370540119003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.066480] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60224}, multi_version_start=1713895370540119003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.067037] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60120}, multi_version_start=1713895370730203000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.067506] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60377}, multi_version_start=1713895370937432003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.068024] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50398}, multi_version_start=1713895370937432003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.068444] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60115}, multi_version_start=1713895370937432003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.069005] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50259}, multi_version_start=1713895370937432003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.069493] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60242}, multi_version_start=1713895371241110003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.070186] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50254}, multi_version_start=1713895371241110003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.070731] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:260}, multi_version_start=1713895371241110003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.071290] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101085}, multi_version_start=1713895371498844000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.071785] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60209}, multi_version_start=1713895371498844000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.072366] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=133] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50335}, multi_version_start=1713895372476130000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.072818] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:241}, multi_version_start=1713895372476130001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.073384] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=139] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:306}, multi_version_start=1713895372644270001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.073960] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60443}, multi_version_start=1713895372644270001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.074525] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:132}, multi_version_start=1713895372910834003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.075106] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101087}, multi_version_start=1713895372910834003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.075724] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50153}, multi_version_start=1713895372910834003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.076310] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60490}, multi_version_start=1713895373052973002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.076838] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:336}, multi_version_start=1713895373052973003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.077298] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50233}, multi_version_start=1713895373052973003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.077799] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50427}, multi_version_start=1713895373052973003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.078275] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50191}, multi_version_start=1713895373348689000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.078809] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:343}, multi_version_start=1713895373348689000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.079277] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60232}, multi_version_start=1713895373348689002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.079771] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101051}, multi_version_start=1713895373348689002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.080447] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=260] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:100006}, multi_version_start=1713895373348689002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.081008] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60382}, multi_version_start=1713895373348689002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.081517] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60396}, multi_version_start=1713895373516795001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.082069] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60237}, multi_version_start=1713895373516795002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.082932] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=443] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101061}, multi_version_start=1713895373516795002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.083571] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=282] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60202}, multi_version_start=1713895373516795002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.084093] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=118] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50181}, multi_version_start=1713895373516795002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.084596] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=124] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60211}, multi_version_start=1713895374754197001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.085116] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50241}, multi_version_start=1713895374754197001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.085181] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [862344][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=62] ====== tenant freeze timer task ====== [2024-04-24 10:00:47.085317] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [862344][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=104] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=96468992, ctx.total_memstore_used_=130023424, ctx.total_memstore_hold_=130023424, memstore_freeze_trigger_limit_=107374180, tenant_id=1001) [2024-04-24 10:00:47.085404] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [862344][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=74] [TenantFreezer] oldest frozen memtable(list="") [2024-04-24 10:00:47.085656] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101076}, multi_version_start=1713895375089680000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.086130] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50168}, multi_version_start=1713895375089680000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.086657] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1713920414963170000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.087193] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50107}, multi_version_start=1713895375244330000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.087702] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101006}, multi_version_start=1713895375415608001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.087904] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [874414][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=75] LogLoopThread round_cost_time(us)(round_cost_time=99) [2024-04-24 10:00:47.088163] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60272}, multi_version_start=1713895375415608001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.088775] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:250}, multi_version_start=1713895375415608001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.089335] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=165] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50222}, multi_version_start=1713895375555497003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.089928] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60114}, multi_version_start=1713895375766632000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.090442] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50283}, multi_version_start=1713895375766632000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.091003] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=134] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:425}, multi_version_start=1713895375766632000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.091483] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60221}, multi_version_start=1713895375766632000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.092078] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=107] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50247}, multi_version_start=1713895375910282001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.092648] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101074}, multi_version_start=1713895375910282001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.093156] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60192}, multi_version_start=1713895376063529000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.093588] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50323}, multi_version_start=1713895376063529000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.093649] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [2320947][T1004_L0_G0][T1004][YB420A000003-000616C2C0D8F348-0-0] [lt=115] current clock is (clock_=0, max_seq_=0, mem_limit=3181127926, cur_mem_hold=297795584, attr_.tenant_id_=1004) [2024-04-24 10:00:47.094158] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101053}, multi_version_start=1713895376063529000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.094604] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:211}, multi_version_start=1713895376173900000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.095153] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50493}, multi_version_start=1713895376173900000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.095535] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [2320947][T1004_L0_G0][T1004][YB420A000003-000616C2C0D8F349-0-0] [lt=115] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, append size=838) [2024-04-24 10:00:47.095606] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50105}, multi_version_start=1713895376173900000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.095666] INFO [LIB] stat (utility.h:1197) [2320947][T1004_L0_G0][T1004][YB420A000003-000616C2C0D8F349-0-0] [lt=108] [PALF STAT APPEND COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=0, this=0x14b0c0f0ae50, extra_info="id:1001") [2024-04-24 10:00:47.096167] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=104] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50237}, multi_version_start=1713895376500826000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.096684] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60142}, multi_version_start=1713895376760379000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.097198] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50130}, multi_version_start=1713895376760379000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.097655] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:398}, multi_version_start=1713895376912610000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.098320] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50352}, multi_version_start=1713895376912610002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.098759] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60107}, multi_version_start=1713895376912610002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.099315] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50112}, multi_version_start=1713895376912610002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.099395] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=75] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.099500] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=86] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.099610] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=86] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.099927] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=53] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:491}, multi_version_start=1713895377922800002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.100540] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60474}, multi_version_start=1713895377922800002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.101123] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50405}, multi_version_start=1713895378123307002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.101558] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60152}, multi_version_start=1713895378285092000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.101931] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:218) [861386][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=71] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-04-24 10:00:47.102077] INFO [COMMON] handle (memory_dump.cpp:498) [861386][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=88] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-04-24 10:00:47.102160] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=104] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60291}, multi_version_start=1713895378285092000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.102345] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [861386][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=117] update_check_range(min_check_version=4848, max_check_version=4849, global_version=4850) [2024-04-24 10:00:47.102695] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60428}, multi_version_start=1713895378418601001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.103319] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50102}, multi_version_start=1713895378418601001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.103925] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60102}, multi_version_start=1713895378418601001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.104279] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=90] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8892784640, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:47.104447] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:431}, multi_version_start=1713895378587323001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.105017] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60295}, multi_version_start=1713895378587323002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.105603] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=106] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:112}, multi_version_start=1713895378587323002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.106240] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=133] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60241}, multi_version_start=1713895378687230000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.106767] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50297}, multi_version_start=1713895382358946001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.107310] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:437}, multi_version_start=1713895382358946001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.107829] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60208}, multi_version_start=1713895382358946001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.108383] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=142] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60434}, multi_version_start=1713895382358946001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.108934] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60169}, multi_version_start=1713895383394528000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.109385] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101042}, multi_version_start=1713895383394528000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.109930] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=107] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:145}, multi_version_start=1713895383394528000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.110383] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50328}, multi_version_start=1713895383394528000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.111008] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=160] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50224}, multi_version_start=1713895383650958000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.111580] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=123] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:222}, multi_version_start=1713895383650958000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.112109] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60190}, multi_version_start=1713895383650958000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.112608] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50293}, multi_version_start=1713895383650958000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.113136] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:115}, multi_version_start=1713895383810720003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.113583] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60247}, multi_version_start=1713895383810720003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.114135] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=154] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:352}, multi_version_start=1713895383810720003, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.114702] INFO [LIB] stat (utility.h:1197) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=129] [PALF STAT WRITE LOG COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=283824, this=0x14b1470b53b0, extra_info="palf_id:1") [2024-04-24 10:00:47.114849] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=128][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67158053}], scn_array=[{val:1713924046512037000, v:0}], curr_size=498, accum_size=1240, time_cost=539838) [2024-04-24 10:00:47.114966] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=110] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=1240) [2024-04-24 10:00:47.115064] INFO [LIB] stat (utility.h:1214) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=69] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=183, this=0x14b1679f6530, extra_info="") [2024-04-24 10:00:47.115240] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=99] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=63, this=0x14b1470ab470, extra_info="") [2024-04-24 10:00:47.115329] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:242}, multi_version_start=1713895383958369002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.115350] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=76] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=4, avg_log_batch_cnt=1, total_group_log_size=920, avg_group_log_size=230) [2024-04-24 10:00:47.115490] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=76][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=259311, log_task={header:{begin_lsn:{lsn:67158053}, end_lsn:{lsn:67158551}, log_id:259311, min_scn:{val:1713924046512037000, v:0}, max_scn:{val:1713924046512037000, v:0}, data_len:442, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:67157931}, data_checksum:315985663, accum_checksum:2413219968, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924046512703, freeze_ts:1713924046574460, submit_ts:1713924046574577, flushed_ts:1713924047115120, gen_to_freeze cost time:61757, gen_to_submit cost time:61874, submit_to_flush cost time:540543}, fs_cb_begin_ts=1713924047115442, log_life_time=602739) [2024-04-24 10:00:47.115671] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=83] [PALF STAT FLUSH CB COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=325, this=0x14b1470b55e0, extra_info="palf_id:1") [2024-04-24 10:00:47.115662] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=92][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924046512037000, v:0}, log_ts:{val:1713924046512037000, v:0}, lsn:{lsn:67158109}, submit_ts:1713924046512037}, this:0x14b168f7bb00, is_inited_:true, trans_id:{txid:869501}, ls_id:{id:1}, ctx:0x14b168f78b90, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=603573, ctx_lock_wait_time=42) [2024-04-24 10:00:47.115935] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50499}, multi_version_start=1713895383958369002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.116604] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:440}, multi_version_start=1713895383958369002, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.117119] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50197}, multi_version_start=1713895384191449001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.117615] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50148}, multi_version_start=1713895384191449001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.118227] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=128] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60222}, multi_version_start=1713895384191449001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.118745] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60290}, multi_version_start=1713895384191449001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.119325] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101038}, multi_version_start=1713895384701500000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.119765] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60260}, multi_version_start=1713895384701500000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.120317] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50169}, multi_version_start=1713895384916312000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.120790] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60238}, multi_version_start=1713895384916312000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.121373] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=122] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101090}, multi_version_start=1713895384916312000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.122000] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=111] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60328}, multi_version_start=1713895384916312000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.122475] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:442}, multi_version_start=1713895385066465001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.123032] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:373}, multi_version_start=1713899298318246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.124519] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=64][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.127150] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C38-0-0] [lt=76] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=742) [2024-04-24 10:00:47.140815] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=110] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.141197] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [874742][T1003_LSArchive][T1003][YB420A000003-000616C2B608F0E7-0-0] [lt=81] gc stale ls task succ [2024-04-24 10:00:47.143004] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [874587][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A3-0-0] [lt=114][errcode=-4023] tx is 2pc logging(this={this:0x14b055e1a050, ref:3, trans_id:{txid:176904}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138442, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589701], ctx_create_time:1713924046585672}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046585674, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452001, v:0}, log_ts:{val:1713924046589452001, v:0}, lsn:{lsn:79580890}, submit_ts:1713924046589452}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176904}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176904}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452001, v:0}, start_scn:{val:1713924046589452001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877457407528 trans_version={val:1713924046589452001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176904} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3140423830 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:47.143163] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [874587][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A3-0-0] [lt=145][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e1a050, ref:3, trans_id:{txid:176904}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138442, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589701], ctx_create_time:1713924046585672}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046585674, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452001, v:0}, log_ts:{val:1713924046589452001, v:0}, lsn:{lsn:79580890}, submit_ts:1713924046589452}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176904}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176904}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452001, v:0}, start_scn:{val:1713924046589452001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877457407528 trans_version={val:1713924046589452001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176904} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3140423830 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:47.143286] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [874587][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A3-0-0] [lt=110][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176904}) [2024-04-24 10:00:47.143394] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [874587][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A3-0-0] [lt=99][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176904}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924047138527, cluster_id:1713874979}, expire_ts:1713925045138403, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.143527] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [874587][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A3-0-0] [lt=115][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176904}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924047138527, cluster_id:1713874979}, expire_ts:1713925045138403, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.144204] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2628-0-0] [lt=135][errcode=-4023] tx is 2pc logging(this={this:0x14b0fbedf050, ref:3, trans_id:{txid:338711}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924076767546, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046792910], ctx_create_time:1713924046778243}{ls_id:{id:1}, session_id:3221795357, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046789276, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046791807000, v:0}, log_ts:{val:1713924046791807000, v:0}, lsn:{lsn:298513878}, submit_ts:1713924046792973}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:338711}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338711}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046791807000, v:0}, start_scn:{val:1713924046791807000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924046791807000, v:0} commit_version={val:1713924046791807000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338711} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=215183838 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046791807000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046792973, block_frozen_memtable:null}) [2024-04-24 10:00:47.144397] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2628-0-0] [lt=183][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fbedf050, ref:3, trans_id:{txid:338711}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924076767546, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046792910], ctx_create_time:1713924046778243}{ls_id:{id:1}, session_id:3221795357, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046789276, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046791807000, v:0}, log_ts:{val:1713924046791807000, v:0}, lsn:{lsn:298513878}, submit_ts:1713924046792973}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:338711}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338711}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046791807000, v:0}, start_scn:{val:1713924046791807000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924046791807000, v:0} commit_version={val:1713924046791807000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338711} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=215183838 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046791807000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046792973, block_frozen_memtable:null}) [2024-04-24 10:00:47.144530] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2628-0-0] [lt=125][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338711}) [2024-04-24 10:00:47.144646] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2628-0-0] [lt=104][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338711}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047137923, cluster_id:1713874979}, expire_ts:1713924076788074, commit_start_scn:{val:1713924046780604000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.144720] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2628-0-0] [lt=61][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338711}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047137923, cluster_id:1713874979}, expire_ts:1713924076788074, commit_start_scn:{val:1713924046780604000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.148451] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [862309][T1001_Occam][T1001][YB420A000003-000616C2AFF988F0-0-0] [lt=70] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:47.155364] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=84][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.156977] INFO handle (memory_dump.cpp:626) [861386][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=88] statistics: tenant_cnt: 7, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 110 211 0 1 2 62 1 0 1 4 1 1 0 1 5 41 5 0 1 7 1 2 0 1 8 56 1 0 1 12 8 3 0 1 13 5 1 0 1 14 1 1 0 1 15 1 1 0 1 16 2 1 0 1 18 12 3 0 1 20 1 1 0 500 0 82 244 0 500 7 6 3 0 500 8 42 1 0 500 9 2 1 0 500 10 6 2 0 500 16 3 1 0 500 17 8 7 0 500 22 92 61 0 500 23 50 11 0 508 0 6 13 0 508 8 13 1 0 1001 0 90 212 0 1001 2 62 1 0 1001 4 1 1 0 1001 5 23 5 0 1001 8 43 1 0 1001 12 6 3 0 1001 13 3 1 0 1001 14 1 2 0 1001 16 1 1 0 1001 18 12 3 0 1001 20 1 1 0 1002 0 110 218 0 1002 2 52 1 0 1002 4 2 1 0 1002 5 23 5 0 1002 8 49 1 0 1002 12 14 3 0 1002 13 4 1 0 1002 14 1 2 0 1002 18 16 3 0 1002 20 2 1 0 1003 0 93 213 0 1003 2 58 1 0 1003 4 1 1 0 1003 5 26 5 0 1003 8 43 1 0 1003 12 6 3 0 1003 13 5 1 0 1003 14 1 2 0 1003 16 1 1 0 1003 18 10 3 0 1003 20 1 1 0 1004 0 190 243 0 1004 2 142 1 0 1004 4 2 1 0 1004 5 218 6 0 1004 7 2 2 0 1004 8 61 1 0 1004 12 9 3 0 1004 13 5 1 0 1004 14 1 2 0 1004 15 1 1 0 1004 16 1 1 0 1004 18 11 3 0 1004 20 2 1 0 1004 21 1 1 0 cost_time: 54645 [2024-04-24 10:00:47.164785] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=197] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.164909] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=116] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.199572] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=175] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.199728] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=107] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.199836] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=99] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.220008] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [874754][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=106] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:47.220040] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861495][RpcIO][T0][Y0-0000000000000000-0-0] [lt=107] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:47.220938] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861497][RpcIO][T0][Y0-0000000000000000-0-0] [lt=105] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:47.221012] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861496][RpcIO][T0][Y0-0000000000000000-0-0] [lt=105] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:47.221732] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861498][RpcIO][T0][Y0-0000000000000000-0-0] [lt=111] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:47.224719] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=118][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.225004] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [2320947][T1004_L0_G0][T1004][YB420A000003-000616C2C0D8F34F-0-0] [lt=112] current clock is (clock_=0, max_seq_=0, mem_limit=3181127926, cur_mem_hold=297795584, attr_.tenant_id_=1004) [2024-04-24 10:00:47.232792] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=74] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=975136, replace_num=15728) [2024-04-24 10:00:47.232957] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=156] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-04-24 10:00:47.234469] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:231) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=95] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"10.0.0.3:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3}}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2"], tenant_id:1, replica_type:0}}, {unit:unit_id:1002, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"10.0.0.3:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"ocp_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"2GB", log_disk_size:"6GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:1}}, pool:{resource_pool_id:1001, name:"ocp_pool", unit_count:1, unit_config_id:1001, zone_list:["zone2", "zone1"], tenant_id:1002, replica_type:0}}, {unit:unit_id:1003, resource_pool_id:1005, unit_group_id:1002, zone:"zone1", server:"10.0.0.3:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1004, name:"c8", resource:{min_cpu:2, max_cpu:8, memory_size:"12GB", log_disk_size:"36GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:2}}, pool:{resource_pool_id:1005, name:"c8p", unit_count:1, unit_config_id:1004, zone_list:["zone1", "zone2"], tenant_id:1004, replica_type:0}}]) [2024-04-24 10:00:47.234656] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:810) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=144] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2024-04-24 10:00:47.234756] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:658) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=88] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.6GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:1}, user_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"5.4GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:1}, unit_resource={min_cpu:1, max_cpu:1, memory_size:"2GB", log_disk_size:"6GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:1}) [2024-04-24 10:00:47.234837] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:125) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=72] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1002, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"ocp_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.6GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:1}}, mode:0, create_timestamp:0, is_removed:false}, user_config={tenant_id:1002, unit_id:1002, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"ocp_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"5.4GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:1}}, mode:0, create_timestamp:0, is_removed:false}) [2024-04-24 10:00:47.235007] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:810) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=149] jx_debug: get tenant compatibility mode(tenant_id=1004, compat_mode=0) [2024-04-24 10:00:47.235074] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:658) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=59] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1.2GB", log_disk_size:"3.6GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:2}, user_resource={min_cpu:2, max_cpu:8, memory_size:"10.8GB", log_disk_size:"32.4GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:2}, unit_resource={min_cpu:2, max_cpu:8, memory_size:"12GB", log_disk_size:"36GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:2}) [2024-04-24 10:00:47.235181] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:125) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=97] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1003, unit_id:1003, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1004, name:"c8", resource:{min_cpu:1, max_cpu:1, memory_size:"1.2GB", log_disk_size:"3.6GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:2}}, mode:0, create_timestamp:0, is_removed:false}, user_config={tenant_id:1004, unit_id:1003, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1004, name:"c8", resource:{min_cpu:2, max_cpu:8, memory_size:"10.8GB", log_disk_size:"32.4GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:2}}, mode:0, create_timestamp:0, is_removed:false}) [2024-04-24 10:00:47.235377] INFO [COMMON] update_io_clock (ob_io_mclock.cpp:454) [861548][OmtNodeBalancer][T1][YB420A000003-000616C2A8291461-0-0] [lt=81] update other group clock success(index=0, unit_config={min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3}, cur_config={deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}) [2024-04-24 10:00:47.235496] INFO [COMMON] update_basic_io_config (ob_io_manager.cpp:918) [861548][OmtNodeBalancer][T1][YB420A000003-000616C2A8291461-0-0] [lt=112] update basic io config success(tenant_id_=1, io_config_={group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, io_config={group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:0, max_percent:0, weight_percent:0}]}, io_clock_={is_inited_:true, group_clocks:[], other_clock:{is_inited_:true, is_stopped_:false, reservation_clock:{iops:9223372036854775807, last_ns:0}, is_unlimited:false, limitation_clock:{iops:9223372036854775807, last_ns:0}, proportion_clock:{iops:3, last_ns:0}}, unit_clock:{iops:9223372036854775807, last_ns:0}, io_config_:{group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, io_usage_:{doing_request_count:[0:0]}}) [2024-04-24 10:00:47.235687] INFO [COMMON] update_io_clock (ob_io_mclock.cpp:454) [861548][OmtNodeBalancer][T1002][YB420A000003-000616C2A8291461-0-0] [lt=101] update other group clock success(index=0, unit_config={min_iops:9223372036854775807, max_iops:9223372036854775807, weight:1}, cur_config={deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}) [2024-04-24 10:00:47.235789] INFO [COMMON] update_basic_io_config (ob_io_manager.cpp:918) [861548][OmtNodeBalancer][T1002][YB420A000003-000616C2A8291461-0-0] [lt=94] update basic io config success(tenant_id_=1002, io_config_={group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:1}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, io_config={group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:1}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:0, max_percent:0, weight_percent:0}]}, io_clock_={is_inited_:true, group_clocks:[], other_clock:{is_inited_:true, is_stopped_:false, reservation_clock:{iops:9223372036854775807, last_ns:0}, is_unlimited:false, limitation_clock:{iops:9223372036854775807, last_ns:0}, proportion_clock:{iops:1, last_ns:0}}, unit_clock:{iops:9223372036854775807, last_ns:0}, io_config_:{group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:1}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, io_usage_:{doing_request_count:[0:1]}}) [2024-04-24 10:00:47.235968] INFO [COMMON] update_io_clock (ob_io_mclock.cpp:454) [861548][OmtNodeBalancer][T1001][YB420A000003-000616C2A8291461-0-0] [lt=112] update other group clock success(index=0, unit_config={min_iops:9223372036854775807, max_iops:9223372036854775807, weight:1}, cur_config={deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}) [2024-04-24 10:00:47.236066] INFO [COMMON] update_basic_io_config (ob_io_manager.cpp:918) [861548][OmtNodeBalancer][T1001][YB420A000003-000616C2A8291461-0-0] [lt=89] update basic io config success(tenant_id_=1001, io_config_={group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:1}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, io_config={group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:1}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:0, max_percent:0, weight_percent:0}]}, io_clock_={is_inited_:true, group_clocks:[], other_clock:{is_inited_:true, is_stopped_:false, reservation_clock:{iops:9223372036854775807, last_ns:0}, is_unlimited:false, limitation_clock:{iops:9223372036854775807, last_ns:0}, proportion_clock:{iops:1, last_ns:0}}, unit_clock:{iops:9223372036854775807, last_ns:0}, io_config_:{group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:1}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, io_usage_:{doing_request_count:[0:0]}}) [2024-04-24 10:00:47.236274] INFO [COMMON] update_io_clock (ob_io_mclock.cpp:454) [861548][OmtNodeBalancer][T1004][YB420A000003-000616C2A8291461-0-0] [lt=133] update other group clock success(index=0, unit_config={min_iops:9223372036854775807, max_iops:9223372036854775807, weight:2}, cur_config={deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}) [2024-04-24 10:00:47.236345] INFO [COMMON] update_basic_io_config (ob_io_manager.cpp:918) [861548][OmtNodeBalancer][T1004][YB420A000003-000616C2A8291461-0-0] [lt=65] update basic io config success(tenant_id_=1004, io_config_={group_num_:0, memory_limit_:11596411700, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:2}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, io_config={group_num_:0, memory_limit_:11596411700, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:2}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:0, max_percent:0, weight_percent:0}]}, io_clock_={is_inited_:true, group_clocks:[], other_clock:{is_inited_:true, is_stopped_:false, reservation_clock:{iops:9223372036854775807, last_ns:0}, is_unlimited:false, limitation_clock:{iops:9223372036854775807, last_ns:0}, proportion_clock:{iops:2, last_ns:0}}, unit_clock:{iops:9223372036854775807, last_ns:0}, io_config_:{group_num_:0, memory_limit_:11596411700, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:2}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, io_usage_:{doing_request_count:[0:1]}}) [2024-04-24 10:00:47.236551] INFO [COMMON] update_io_clock (ob_io_mclock.cpp:454) [861548][OmtNodeBalancer][T1003][YB420A000003-000616C2A8291461-0-0] [lt=129] update other group clock success(index=0, unit_config={min_iops:9223372036854775807, max_iops:9223372036854775807, weight:2}, cur_config={deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}) [2024-04-24 10:00:47.236709] INFO [COMMON] update_basic_io_config (ob_io_manager.cpp:918) [861548][OmtNodeBalancer][T1003][YB420A000003-000616C2A8291461-0-0] [lt=151] update basic io config success(tenant_id_=1003, io_config_={group_num_:0, memory_limit_:1288490188, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:2}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, io_config={group_num_:0, memory_limit_:1288490188, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:2}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:0, max_percent:0, weight_percent:0}]}, io_clock_={is_inited_:true, group_clocks:[], other_clock:{is_inited_:true, is_stopped_:false, reservation_clock:{iops:9223372036854775807, last_ns:0}, is_unlimited:false, limitation_clock:{iops:9223372036854775807, last_ns:0}, proportion_clock:{iops:2, last_ns:0}}, unit_clock:{iops:9223372036854775807, last_ns:0}, io_config_:{group_num_:0, memory_limit_:1288490188, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:2}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, io_usage_:{doing_request_count:[0:0]}}) [2024-04-24 10:00:47.238542] INFO [STORAGE.TRANS] get_gts (ob_gts_source.cpp:268) [861548][OmtNodeBalancer][T1][YB420A000003-000616C2A8291461-0-0] [lt=94] push gts task success(*task={:""}) [2024-04-24 10:00:47.240970] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=147] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.243223] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A4-0-0] [lt=125][errcode=-4023] tx is 2pc logging(this={this:0x14b148003290, ref:4, trans_id:{txid:176905}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924076581982, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589598], ctx_create_time:1713924046582067}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046580909, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452000, v:0}, log_ts:{val:1713924046589452000, v:0}, lsn:{lsn:79580022}, submit_ts:1713924046589452}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176905}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176905}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452000, v:0}, start_scn:{val:1713924046589452000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465665976 max_table_version=1713877465684248 trans_version={val:1713924046589452000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176905} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=924633448 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:47.243443] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A4-0-0] [lt=207][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b148003290, ref:4, trans_id:{txid:176905}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924076581982, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589598], ctx_create_time:1713924046582067}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046580909, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452000, v:0}, log_ts:{val:1713924046589452000, v:0}, lsn:{lsn:79580022}, submit_ts:1713924046589452}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176905}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176905}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452000, v:0}, start_scn:{val:1713924046589452000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465665976 max_table_version=1713877465684248 trans_version={val:1713924046589452000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176905} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=924633448 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:47.243508] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A5-0-0] [lt=91][errcode=-4023] tx is 2pc logging(this={this:0x14b147fffbd0, ref:4, trans_id:{txid:176906}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925046666492, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046708409], ctx_create_time:1713924046701144}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79581303}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046705015, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046707343000, v:0}, log_ts:{val:1713924046707343000, v:0}, lsn:{lsn:79582268}, submit_ts:1713924046708648}, this:0x14b148002b40, is_inited_:true, trans_id:{txid:176906}, ls_id:{id:1}, ctx:0x14b147fffbd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176906}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046707343000, v:0}, start_scn:{val:1713924046589452002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877459227536 max_table_version=1713877465782000 trans_version={val:1713924046707343000, v:0} commit_version={val:1713924046707343000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176906} ls_id=1 callback_alloc_count=5 callback_free_count=0 checksum=3165083175 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=8 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=8 unsynced_cnt=5 unsubmitted_cnt_=0 cb_statistics:[main=8, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046708648, block_frozen_memtable:null}) [2024-04-24 10:00:47.243672] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A4-0-0] [lt=169][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176905}) [2024-04-24 10:00:47.243765] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A4-0-0] [lt=87][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176905}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924047238678, cluster_id:1713874979}, expire_ts:1713924076505566, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.243738] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A5-0-0] [lt=223][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b147fffbd0, ref:4, trans_id:{txid:176906}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925046666492, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046708409], ctx_create_time:1713924046701144}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79581303}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046705015, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046707343000, v:0}, log_ts:{val:1713924046707343000, v:0}, lsn:{lsn:79582268}, submit_ts:1713924046708648}, this:0x14b148002b40, is_inited_:true, trans_id:{txid:176906}, ls_id:{id:1}, ctx:0x14b147fffbd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176906}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046707343000, v:0}, start_scn:{val:1713924046589452002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877459227536 max_table_version=1713877465782000 trans_version={val:1713924046707343000, v:0} commit_version={val:1713924046707343000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176906} ls_id=1 callback_alloc_count=5 callback_free_count=0 checksum=3165083175 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=8 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=8 unsynced_cnt=5 unsubmitted_cnt_=0 cb_statistics:[main=8, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046708648, block_frozen_memtable:null}) [2024-04-24 10:00:47.243960] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A4-0-0] [lt=182][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176905}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924047238678, cluster_id:1713874979}, expire_ts:1713924076505566, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.243988] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A5-0-0] [lt=243][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176906}) [2024-04-24 10:00:47.244088] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A5-0-0] [lt=80][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176906}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047239011, cluster_id:1713874979}, expire_ts:1713925046666035, commit_start_scn:{val:1713924046703599000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.244195] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A5-0-0] [lt=85][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176906}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047239011, cluster_id:1713874979}, expire_ts:1713925046666035, commit_start_scn:{val:1713924046703599000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.250562] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:122) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=97] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2024-04-24 10:00:47.255581] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=73][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.256755] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [874535][T1004_ArcSender][T1004][YB420A000003-000616C2B598F0E7-0-0] [lt=97][errcode=-4018] try free send task failed(ret=-4018) [2024-04-24 10:00:47.260031] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [874525][T1004_ReqMemEvi][T1004][Y0-0000000000000000-0-0] [lt=155] sql audit evict task end(request_manager_->get_tenant_id()=1004, evict_high_mem_level=278313880, evict_high_size_level=90000, evict_batch_count=0, elapse_time=6, size_used=80431, mem_used=131023872) [2024-04-24 10:00:47.262300] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:159) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=140] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2024-04-24 10:00:47.265116] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=89] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.265233] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=97] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.266089] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [874706][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=119] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:769, free_obj_hold_cnt:74, allocator used:3351768, allocator total:3466624}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:72, free_obj_hold_cnt:32, allocator used:6821464, allocator total:6821464}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():154752, full allocator used:1600, full allocator total:220288}, tablets_mem=10508376, tablets_mem_limit=257698020, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:23, free_obj_hold_cnt:163, allocator used:369024, allocator total:392448}, pending_cnt=0, wait_gc_count=0, tablet count=841) [2024-04-24 10:00:47.266300] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [862437][T1001_LSArchive][T1001][YB420A000003-000616C2AEF8F0E7-0-0] [lt=79] gc stale ls task succ [2024-04-24 10:00:47.269182] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:196) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=96] refresh_resource_function_mapping_rule(tenant_id=1, plan=, rules=[]) [2024-04-24 10:00:47.269278] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:100) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=88] refresh resource mapping rule success(tenant_id=1, plan=) [2024-04-24 10:00:47.281540] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:270) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78][errcode=0] cost too much time:mds_table_base.cpp:int oceanbase::storage::mds::MdsTableBase::advance_state_to(oceanbase::storage::mds::MdsTableBase::State) const, (*this=|threshold=1.00ms|start at 10:00:47.279|total=1.83ms) [2024-04-24 10:00:47.292237] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2607-0-0] [lt=124][errcode=-4023] tx is 2pc logging(this={this:0x14b15a16b090, ref:3, trans_id:{txid:391371}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924076773625, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046800195], ctx_create_time:1713924046784446}{ls_id:{id:1}, session_id:3221795340, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046791808, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046799990000, v:0}, log_ts:{val:1713924046799990000, v:0}, lsn:{lsn:314127056}, submit_ts:1713924046799990}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:391371}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:391371}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046799990000, v:0}, start_scn:{val:1713924046799990000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875442714112 max_table_version=1713875442714112 trans_version={val:1713924046799990000, v:0} commit_version={val:1713924046799990000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:391371} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=2612980561 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046799990000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046799990, block_frozen_memtable:null}) [2024-04-24 10:00:47.292461] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2607-0-0] [lt=210][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b15a16b090, ref:3, trans_id:{txid:391371}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924076773625, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046800195], ctx_create_time:1713924046784446}{ls_id:{id:1}, session_id:3221795340, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046791808, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046799990000, v:0}, log_ts:{val:1713924046799990000, v:0}, lsn:{lsn:314127056}, submit_ts:1713924046799990}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:391371}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:391371}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046799990000, v:0}, start_scn:{val:1713924046799990000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875442714112 max_table_version=1713875442714112 trans_version={val:1713924046799990000, v:0} commit_version={val:1713924046799990000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:391371} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=2612980561 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046799990000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046799990, block_frozen_memtable:null}) [2024-04-24 10:00:47.292569] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2607-0-0] [lt=103][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:391371}) [2024-04-24 10:00:47.292640] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2607-0-0] [lt=63][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1001, tx_id:{txid:391371}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047286906, cluster_id:1713874979}, expire_ts:1713924076795547, commit_start_scn:{val:1713924046786976000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.292738] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2607-0-0] [lt=69][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1001, tx_id:{txid:391371}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047286906, cluster_id:1713874979}, expire_ts:1713924076795547, commit_start_scn:{val:1713924046786976000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.294252] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [861452][AutoLSLoc][T0][YB420A000003-000616C2AAD916E7-0-0] [lt=110] fetch root ls success(local_cluster_id=1713874979, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.2:2882", sql_port:2881, role:1, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:5, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713921656566585, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.3:2882", sql_port:2881, role:2, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["10.0.0.2:2882", "10.0.0.3:2882"]) [2024-04-24 10:00:47.299812] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=155] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.299952] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=120] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.300034] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=75] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.305007] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=98] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8892784640, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:47.325006] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=144][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.326919] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [862245][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=134] report RowHolderMapper summary info(count=1, bkt_cnt=248) [2024-04-24 10:00:47.329601] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=90] ====== checkpoint timer task ====== [2024-04-24 10:00:47.329972] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=166] get_max_applied_scn(scn={val:1713924045997782000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:7, palf_committed_end_lsn_:{lsn:298512772}, last_check_scn_:{val:1713924046684930000, v:0}, max_applied_cb_scn_:{val:1713924045997782000, v:0}}) [2024-04-24 10:00:47.330093] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=109] get_min_unreplayed_log_info(lsn={lsn:283486487}, scn={val:1713900515126283548, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713921656870314, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:283486487}, next_to_submit_scn_:{val:1713900515126283548, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x14b12a08d000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:283486487}, end_lsn:{lsn:283486487}, read_buf:{buf_len_:8192, buf_:0x14b12a08d000}, block_size:67104768, log_storage_:0x14b0f0aa7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-04-24 10:00:47.332250] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=133] succ to get rec scn(*this={this:0x14b0f2804030, ls_id:{id:1}, tenant_id:1003, state:"L_WORKING", total_tx_ctx_count:4, active_tx_count:4, total_active_readonly_request_count:1, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1713895200598622000, v:0}, last_push_gc_task_ts_:1713895205297191, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:844}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-24 10:00:47.332348] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=92] tx ctx memtable get rec scn(this={ObITable:{this:0x14b0f0adc080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1713895201189153, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x14b0f0adc080, snapshot_version:{val:1713895201189153, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:1713924046506776000, v:0}) [2024-04-24 10:00:47.332534] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=164] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-04-24 10:00:47.333260] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=193] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1713895315527235001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1713895315527235001, v:0}}, }, *this={this:0x14b0f0a70510, is_inited:true, freezing_scn:{val:1713878494144158000, v:0}, ls:{ls_meta:{tenant_id:1003, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895201246660000, v:0}, clog_base_lsn:{lsn:67104768}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1713877340363486001, v:0}, all_id_meta:{id_meta:[{limited_id:1713924059270392000, latest_log_ts:{val:1713924042654251001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713900544098309001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1713877439572285001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:7, palf_env_:0x14b14219e030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:47.333451] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=170] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x14b0f0a7eb90}, min_rec_scn={val:1713895201246660000, v:0}, ls_id_={id:1}) [2024-04-24 10:00:47.333538] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=74] get rec log scn(service_type_=0, rec_log_ts={val:1713924042654251001, v:0}) [2024-04-24 10:00:47.333613] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=67] get rec log scn(service_type_=1, rec_log_ts={val:1713900544098309001, v:0}) [2024-04-24 10:00:47.333711] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=91] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2024-04-24 10:00:47.333827] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=96] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713895201246660000, v:0}, checkpoint_scn_in_ls_meta={val:1713895201246660000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-04-24 10:00:47.333971] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=127][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, curr_base_lsn={lsn:67104768}, new_base_lsn={lsn:67104768}, lsn={lsn:67104768}) [2024-04-24 10:00:47.334079] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=102] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:67104768}, ls->get_ls_id()={id:1}) [2024-04-24 10:00:47.334153] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=67] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-04-24 10:00:47.336333] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=95] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: MICRO_BLOCK_CACHE_IO, mode: read, size: 4096.00, iops: 11.92, rt: 644.83) [2024-04-24 10:00:47.336468] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=108] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:47.337151] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=116] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 66.00) [2024-04-24 10:00:47.337292] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=118] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:47.339295] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=97] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-24 10:00:47.339421] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=105] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-24 10:00:47.341134] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=78] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.342979] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2629-0-0] [lt=113][errcode=-4023] tx is 2pc logging(this={this:0x14b0fbedf050, ref:3, trans_id:{txid:338711}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924076767546, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046792910], ctx_create_time:1713924046778243}{ls_id:{id:1}, session_id:3221795357, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046789276, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046791807000, v:0}, log_ts:{val:1713924046791807000, v:0}, lsn:{lsn:298513878}, submit_ts:1713924046792973}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:338711}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338711}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046791807000, v:0}, start_scn:{val:1713924046791807000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924046791807000, v:0} commit_version={val:1713924046791807000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338711} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=215183838 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046791807000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046792973, block_frozen_memtable:null}) [2024-04-24 10:00:47.343192] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2629-0-0] [lt=197][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fbedf050, ref:3, trans_id:{txid:338711}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924076767546, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046792910], ctx_create_time:1713924046778243}{ls_id:{id:1}, session_id:3221795357, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046789276, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046791807000, v:0}, log_ts:{val:1713924046791807000, v:0}, lsn:{lsn:298513878}, submit_ts:1713924046792973}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:338711}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338711}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046791807000, v:0}, start_scn:{val:1713924046791807000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924046791807000, v:0} commit_version={val:1713924046791807000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338711} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=215183838 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046791807000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046792973, block_frozen_memtable:null}) [2024-04-24 10:00:47.343341] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2629-0-0] [lt=143][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338711}) [2024-04-24 10:00:47.343421] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2629-0-0] [lt=74][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338711}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047338462, cluster_id:1713874979}, expire_ts:1713924076788074, commit_start_scn:{val:1713924046780604000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.343585] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2629-0-0] [lt=140][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338711}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047338462, cluster_id:1713874979}, expire_ts:1713924076788074, commit_start_scn:{val:1713924046780604000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.344012] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A6-0-0] [lt=71][errcode=-4023] tx is 2pc logging(this={this:0x14b055e1a050, ref:3, trans_id:{txid:176904}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138442, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589701], ctx_create_time:1713924046585672}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046585674, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452001, v:0}, log_ts:{val:1713924046589452001, v:0}, lsn:{lsn:79580890}, submit_ts:1713924046589452}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176904}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176904}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452001, v:0}, start_scn:{val:1713924046589452001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877457407528 trans_version={val:1713924046589452001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176904} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3140423830 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:47.344161] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A6-0-0] [lt=133][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e1a050, ref:3, trans_id:{txid:176904}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138442, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589701], ctx_create_time:1713924046585672}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046585674, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452001, v:0}, log_ts:{val:1713924046589452001, v:0}, lsn:{lsn:79580890}, submit_ts:1713924046589452}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176904}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176904}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452001, v:0}, start_scn:{val:1713924046589452001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877457407528 trans_version={val:1713924046589452001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176904} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3140423830 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:47.344289] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A6-0-0] [lt=117][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176904}) [2024-04-24 10:00:47.344351] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A6-0-0] [lt=54][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176904}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924047338874, cluster_id:1713874979}, expire_ts:1713925045138403, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.344444] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A6-0-0] [lt=69][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176904}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924047338874, cluster_id:1713874979}, expire_ts:1713925045138403, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.348308] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [874735][T1003_ReqMemEvi][T1003][Y0-0000000000000000-0-0] [lt=70] sql audit evict task end(request_manager_->get_tenant_id()=1003, evict_high_mem_level=19327352, evict_high_size_level=90000, evict_batch_count=0, elapse_time=7, size_used=8004, mem_used=14558208) [2024-04-24 10:00:47.350482] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E94-0-0] [lt=75][errcode=-4023] tx is 2pc logging(this={this:0x14b168f2d710, ref:3, trans_id:{txid:877456}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924646597291, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046606334], ctx_create_time:1713924046597291}{ls_id:{id:1001}, session_id:3221852891, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046603230, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046605790000, v:0}, log_ts:{val:1713924046605790000, v:0}, lsn:{lsn:1002185585}, submit_ts:1713924046605790}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877456}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877456}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046605790000, v:0}, start_scn:{val:1713924046605790000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924046605790000, v:0} commit_version={val:1713924046605790000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877456} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=737308951 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046605790000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046605790, block_frozen_memtable:null}) [2024-04-24 10:00:47.350724] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E94-0-0] [lt=234][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b168f2d710, ref:3, trans_id:{txid:877456}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924646597291, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046606334], ctx_create_time:1713924046597291}{ls_id:{id:1001}, session_id:3221852891, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046603230, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046605790000, v:0}, log_ts:{val:1713924046605790000, v:0}, lsn:{lsn:1002185585}, submit_ts:1713924046605790}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877456}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877456}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046605790000, v:0}, start_scn:{val:1713924046605790000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924046605790000, v:0} commit_version={val:1713924046605790000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877456} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=737308951 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046605790000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046605790, block_frozen_memtable:null}) [2024-04-24 10:00:47.350918] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E94-0-0] [lt=188][errcode=-4023] commit fail(ret=-4023, coord={id:1001}, tx_id={txid:877456}) [2024-04-24 10:00:47.351011] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E94-0-0] [lt=81][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877456}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924047345926, cluster_id:1713874979}, expire_ts:1713924558320000, commit_start_scn:{val:1713924046594982000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:47.351091] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E94-0-0] [lt=63][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877456}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924047345926, cluster_id:1713874979}, expire_ts:1713924558320000, commit_start_scn:{val:1713924046594982000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:47.355791] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=118][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.365312] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=87] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.365413] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=91] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.365503] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:834) [862160][T1002_BalanceEx][T1002][YB420A000003-000616C2B3D9AD9E-0-0] [lt=127] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2024-04-24 10:00:47.365598] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:171) [862160][T1002_BalanceEx][T1002][YB420A000003-000616C2B3D9AD9E-0-0] [lt=89] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2024-04-24 10:00:47.372245] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [874741][T1003_TabletShe][T1003][Y0-0000000000000000-0-0] [lt=96] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-04-24 10:00:47.372321] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=159] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200589}, multi_version_start=1713895385440998000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.372348] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [874741][T1003_TabletShe][T1003][Y0-0000000000000000-0-0] [lt=68] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=9343) [2024-04-24 10:00:47.372776] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201365}, multi_version_start=1713895385440998000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.373362] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=207] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200929}, multi_version_start=1713895385440998000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.373782] INFO [BALANCE.TRANSFER] do_work (ob_tenant_transfer_service.cpp:141) [862225][T1002_TntTransf][T1002][YB420A000003-000616C2B30903DC-0-0] [lt=88] [TENANT_TRANSFER] finish one round(ret=0, ret="OB_SUCCESS", all_tasks_count=0, thread_task_count=0, thread_idx=0) [2024-04-24 10:00:47.373987] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200529}, multi_version_start=1713895386246298000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.374450] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200197}, multi_version_start=1713895386246298002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.375079] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200127}, multi_version_start=1713895386443363000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.375513] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200251}, multi_version_start=1713895386443363000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.376135] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201051}, multi_version_start=1713895386883002000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.376674] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200767}, multi_version_start=1713895386883002000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.377272] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201463}, multi_version_start=1713895386883002000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.377775] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200999}, multi_version_start=1713895387572813001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.378267] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201235}, multi_version_start=1713895387572813001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.383857] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:270) [861558][T1_Occam][T1][YB420A000003-000616C2A92989A4-0-0] [lt=76][errcode=0] cost too much time:mds_table_handler.cpp:int oceanbase::storage::mds::ObMdsTableHandler::try_release_nodes_below(const share::SCN &), (*this=|threshold=5.00ms|start at 10:00:47.378|157=7us|161=5.20ms|total=5.38ms) [2024-04-24 10:00:47.384025] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201309}, multi_version_start=1713895388449421003, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.384043] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:270) [861558][T1_Occam][T1][YB420A000003-000616C2A92989A4-0-0] [lt=168][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::try_recycle_mds_table_(oceanbase::storage::ObTablet &, const share::SCN &), (*this=|threshold=5.00ms|start at 10:00:47.378|468=15us|total=5.49ms) [2024-04-24 10:00:47.384483] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200325}, multi_version_start=1713895388659263000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.385012] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201025}, multi_version_start=1713895388659263000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.385422] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200971}, multi_version_start=1713895388717594002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.385913] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201081}, multi_version_start=1713895388717594002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.386331] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200347}, multi_version_start=1713895388717594004, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.386743] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201151}, multi_version_start=1713895388717594004, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.387292] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=115] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200913}, multi_version_start=1713895388717594004, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.387771] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201483}, multi_version_start=1713895389992852002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.388273] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=155] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201351}, multi_version_start=1713895389992852002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.388712] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201205}, multi_version_start=1713895390122768003, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.389200] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201335}, multi_version_start=1713895390122768003, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.389629] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200911}, multi_version_start=1713895390348851000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.390102] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200193}, multi_version_start=1713895390348851000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.390503] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201239}, multi_version_start=1713895390714357004, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.390941] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=51] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200301}, multi_version_start=1713895390714357007, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.391378] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200703}, multi_version_start=1713895391010940001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.391844] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200379}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.392387] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=155] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201375}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.392912] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201449}, multi_version_start=1713895391240940001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.393365] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200052}, multi_version_start=1713895391240940001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.393804] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200063}, multi_version_start=1713895391407454002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.394302] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200707}, multi_version_start=1713895391407454002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.394805] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201093}, multi_version_start=1713895391562980002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.395025] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:783) [861453][AutoLSLocRpc][T0][YB420A000003-000616C2AAA9AEFF-0-0] [lt=155] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["10.0.0.2:2882", "10.0.0.3:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713874979}, location:{server:"10.0.0.2:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:5}}, {key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1713874979}, location:{server:"10.0.0.2:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:5}}, {key:{tenant_id:1004, ls_id:{id:1002}, cluster_id:1713874979}, location:{server:"10.0.0.2:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:5}}, {key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1713874979}, location:{server:"10.0.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:5}}, {key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1713874979}, location:{server:"10.0.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:5}}, {key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1713874979}, location:{server:"10.0.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:7}}, {key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1713874979}, location:{server:"10.0.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:7}}, {key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1713874979}, location:{server:"10.0.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:5}}, {key:{tenant_id:1004, ls_id:{id:1001}, cluster_id:1713874979}, location:{server:"10.0.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:7}}]) [2024-04-24 10:00:47.395364] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200893}, multi_version_start=1713895391562980002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.395836] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200275}, multi_version_start=1713895391562980003, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.396328] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201431}, multi_version_start=1713895391854189002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.396748] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200725}, multi_version_start=1713895391854189002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.397315] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201135}, multi_version_start=1713895392473036000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.397768] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201487}, multi_version_start=1713895392552004000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.398282] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201095}, multi_version_start=1713895392552004001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.398737] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200355}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.399343] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200739}, multi_version_start=1713895392552004002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.399755] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200799}, multi_version_start=1713895392689340000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.399990] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=66] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.400073] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=67] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.400142] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=60] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.400219] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200691}, multi_version_start=1713895392689340000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.400647] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=54] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201289}, multi_version_start=1713895394069955002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.401084] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201331}, multi_version_start=1713895394069955003, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.401521] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201039}, multi_version_start=1713895394069955003, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.402053] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200867}, multi_version_start=1713895394306177001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.402455] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200271}, multi_version_start=1713895394306177001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.403053] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=106] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200533}, multi_version_start=1713895394414355004, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.403428] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200747}, multi_version_start=1713895394575820001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.403921] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200163}, multi_version_start=1713895394575820001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.404357] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200183}, multi_version_start=1713895394760274000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.404915] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:1152921504606847002}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.405348] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201115}, multi_version_start=1713895394866213001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.405491] INFO [SERVER] reload_tenant_task (ob_tenant_tablet_ttl_mgr.cpp:1297) [862594][T1002_TTLTablet][T1002][YB420A000003-000616C2B42903D0-0-0] [lt=78] finish reload tenant task(local_tenant_task_={tenant_id:1002, task_id:-1, is_usr_trigger:false, need_check:false, is_dirty:false, ttl_continue:true, rsp_time:-1, state:4, is_reused:true}, tenant_task={gmt_create:0, gmt_modified:0, tenant_id:18446744073709551615, table_id:18446744073709551615, tablet_id:18446744073709551615, task_id:0, task_start_time:0, task_update_time:0, trigger_type:0, status:19, ttl_del_cnt:0, max_version_del_cnt:0, scan_cnt:0, row_key:"", ret_code:"OB_SUCCESS"}, is_leader=true) [2024-04-24 10:00:47.405839] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200731}, multi_version_start=1713895394866213001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.406317] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=129] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200997}, multi_version_start=1713895395151350001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.406790] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201429}, multi_version_start=1713895395151350001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.407257] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=107] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200391}, multi_version_start=1713895395793937000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.407736] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200059}, multi_version_start=1713895395793937000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.408337] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200951}, multi_version_start=1713895396304354000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.408821] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200145}, multi_version_start=1713895396304354000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.409381] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=188] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200387}, multi_version_start=1713895396390835000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.409819] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200995}, multi_version_start=1713895396390835000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.410290] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=174] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200199}, multi_version_start=1713895396686334000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.410698] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200677}, multi_version_start=1713895396686334000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.411226] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200579}, multi_version_start=1713895396686334000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.411687] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200493}, multi_version_start=1713895396879683000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.411908] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=125] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1713924047411812], gts:1713924047411182000, latest_srr:[mts=1713924047411812]}) [2024-04-24 10:00:47.412175] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201145}, multi_version_start=1713895396879683001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.412642] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200755}, multi_version_start=1713895397003195000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.413177] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:1152921504606846983}, multi_version_start=1713895397003195000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.413585] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:1152921504606846990}, multi_version_start=1713895397003195000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.414105] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200975}, multi_version_start=1713895397182203000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.414659] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200967}, multi_version_start=1713895397182203000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.415164] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200759}, multi_version_start=1713895397182203000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.415638] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200815}, multi_version_start=1713895397374782000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.415831] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [861702][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=136] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:47.416177] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200745}, multi_version_start=1713895397374782001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.416706] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201439}, multi_version_start=1713895397374782001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.417122] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200645}, multi_version_start=1713895397523334000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.417612] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201279}, multi_version_start=1713895397523334000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.418207] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=120] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200171}, multi_version_start=1713895397523334000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.418791] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200545}, multi_version_start=1713895397523334000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.419353] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201209}, multi_version_start=1713895397860627000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.419822] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200699}, multi_version_start=1713895397860627002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.420331] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:1152921504606846986}, multi_version_start=1713895397860627002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.420803] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201471}, multi_version_start=1713895398122478000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.421215] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200663}, multi_version_start=1713895398122478000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.421646] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201165}, multi_version_start=1713895398273307000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.422110] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200537}, multi_version_start=1713895398446927000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.422508] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200491}, multi_version_start=1713895398446927000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.423006] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200781}, multi_version_start=1713895398673744000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.423398] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200283}, multi_version_start=1713895398673744000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.423797] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200959}, multi_version_start=1713895398790190000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.424246] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200925}, multi_version_start=1713895398836150000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.424734] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200235}, multi_version_start=1713895398836150000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.425114] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=78][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.425313] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201327}, multi_version_start=1713895398990769001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.425848] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200909}, multi_version_start=1713895399040014000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.426614] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=160] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201409}, multi_version_start=1713895399040014000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.427084] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201445}, multi_version_start=1713895399040014000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.427603] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201387}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.428108] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200643}, multi_version_start=1713895399136546001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.428713] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200779}, multi_version_start=1713895399212079000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.429235] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200887}, multi_version_start=1713895399212079000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.429805] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200523}, multi_version_start=1713895399212079000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.430307] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=124] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201021}, multi_version_start=1713895399212079000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.430808] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200215}, multi_version_start=1713895399511242000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.431307] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=124] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201413}, multi_version_start=1713895399511242001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.431826] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200009}, multi_version_start=1713895399511242001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.432288] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=126] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200557}, multi_version_start=1713895399511242001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.432979] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200253}, multi_version_start=1713895399691385000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.433271] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=82] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-04-24 10:00:47.433422] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200875}, multi_version_start=1713895399691385000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.433927] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201317}, multi_version_start=1713895399799321000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.434403] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201459}, multi_version_start=1713895399799321000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.435023] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200385}, multi_version_start=1713895399918622000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.435557] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200665}, multi_version_start=1713895399998116000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.436228] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200337}, multi_version_start=1713895399998116000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.436701] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200711}, multi_version_start=1713895400193647000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.437304] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201097}, multi_version_start=1713895400193647000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.437726] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200363}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.438248] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200751}, multi_version_start=1713895400323038000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.438721] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200565}, multi_version_start=1713895400380233001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.439228] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201373}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.439682] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200811}, multi_version_start=1713895400544684001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.440187] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201181}, multi_version_start=1713895400633466000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.440586] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200659}, multi_version_start=1713895400633466000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.440986] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [874628][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=135] LogLoopThread round_cost_time(us)(round_cost_time=51) [2024-04-24 10:00:47.441115] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200869}, multi_version_start=1713895400633466000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.441343] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=80] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.441575] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200717}, multi_version_start=1713895400771018000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.442086] INFO [BALANCE.TRANSFER] do_work (ob_tenant_transfer_service.cpp:141) [874490][T1004_TntTransf][T1004][YB420A000003-000616C2BA590319-0-0] [lt=101] [TENANT_TRANSFER] finish one round(ret=0, ret="OB_SUCCESS", all_tasks_count=0, thread_task_count=0, thread_idx=0) [2024-04-24 10:00:47.442165] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201281}, multi_version_start=1713895400824382000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.442652] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200801}, multi_version_start=1713895400824382000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.443202] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200503}, multi_version_start=1713895400824382000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.443651] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200605}, multi_version_start=1713895400997949000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.444218] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200743}, multi_version_start=1713895400997949000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.444285] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A7-0-0] [lt=63][errcode=-4023] tx is 2pc logging(this={this:0x14b147fffbd0, ref:4, trans_id:{txid:176906}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925046666492, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046708409], ctx_create_time:1713924046701144}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79581303}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046705015, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046707343000, v:0}, log_ts:{val:1713924046707343000, v:0}, lsn:{lsn:79582268}, submit_ts:1713924046708648}, this:0x14b148002b40, is_inited_:true, trans_id:{txid:176906}, ls_id:{id:1}, ctx:0x14b147fffbd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176906}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046707343000, v:0}, start_scn:{val:1713924046589452002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877459227536 max_table_version=1713877465782000 trans_version={val:1713924046707343000, v:0} commit_version={val:1713924046707343000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176906} ls_id=1 callback_alloc_count=5 callback_free_count=0 checksum=3165083175 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=8 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=8 unsynced_cnt=5 unsubmitted_cnt_=0 cb_statistics:[main=8, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046708648, block_frozen_memtable:null}) [2024-04-24 10:00:47.444439] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A8-0-0] [lt=56][errcode=-4023] tx is 2pc logging(this={this:0x14b148003290, ref:4, trans_id:{txid:176905}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924076581982, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589598], ctx_create_time:1713924046582067}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046580909, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452000, v:0}, log_ts:{val:1713924046589452000, v:0}, lsn:{lsn:79580022}, submit_ts:1713924046589452}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176905}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176905}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452000, v:0}, start_scn:{val:1713924046589452000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465665976 max_table_version=1713877465684248 trans_version={val:1713924046589452000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176905} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=924633448 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:47.444530] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A7-0-0] [lt=234][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b147fffbd0, ref:4, trans_id:{txid:176906}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925046666492, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046708409], ctx_create_time:1713924046701144}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79581303}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046705015, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046707343000, v:0}, log_ts:{val:1713924046707343000, v:0}, lsn:{lsn:79582268}, submit_ts:1713924046708648}, this:0x14b148002b40, is_inited_:true, trans_id:{txid:176906}, ls_id:{id:1}, ctx:0x14b147fffbd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176906}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046707343000, v:0}, start_scn:{val:1713924046589452002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877459227536 max_table_version=1713877465782000 trans_version={val:1713924046707343000, v:0} commit_version={val:1713924046707343000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176906} ls_id=1 callback_alloc_count=5 callback_free_count=0 checksum=3165083175 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=8 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=8 unsynced_cnt=5 unsubmitted_cnt_=0 cb_statistics:[main=8, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046708648, block_frozen_memtable:null}) [2024-04-24 10:00:47.444647] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A7-0-0] [lt=110][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176906}) [2024-04-24 10:00:47.444598] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A8-0-0] [lt=147][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b148003290, ref:4, trans_id:{txid:176905}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924076581982, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589598], ctx_create_time:1713924046582067}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046580909, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452000, v:0}, log_ts:{val:1713924046589452000, v:0}, lsn:{lsn:79580022}, submit_ts:1713924046589452}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176905}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176905}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452000, v:0}, start_scn:{val:1713924046589452000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465665976 max_table_version=1713877465684248 trans_version={val:1713924046589452000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176905} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=924633448 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:47.444717] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A7-0-0] [lt=60][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176906}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047439631, cluster_id:1713874979}, expire_ts:1713925046666035, commit_start_scn:{val:1713924046703599000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.444746] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200247}, multi_version_start=1713895400997949000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.444766] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A8-0-0] [lt=140][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176905}) [2024-04-24 10:00:47.444809] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A7-0-0] [lt=65][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176906}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047439631, cluster_id:1713874979}, expire_ts:1713925046666035, commit_start_scn:{val:1713924046703599000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.444845] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A8-0-0] [lt=73][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176905}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924047439976, cluster_id:1713874979}, expire_ts:1713924076505566, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.444969] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A8-0-0] [lt=110][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176905}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924047439976, cluster_id:1713874979}, expire_ts:1713924076505566, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.445259] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200931}, multi_version_start=1713895401178063001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.445783] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861506][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=98] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:47.445810] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200451}, multi_version_start=1713895401300114001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.446027] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861508][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=80] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:47.446080] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861507][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=100] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:47.446336] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200515}, multi_version_start=1713895401300114001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.446981] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861509][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=101] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:47.447060] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200919}, multi_version_start=1713895401374217000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.447417] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861511][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=104] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:47.447582] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200915}, multi_version_start=1713895401548978000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.448125] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200785}, multi_version_start=1713895401548978000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.448196] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861500][BatchIO][T0][Y0-0000000000000000-0-0] [lt=79] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:47.448683] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200597}, multi_version_start=1713895401646816000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.449122] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861502][BatchIO][T0][Y0-0000000000000000-0-0] [lt=87] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:47.449283] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=131] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201229}, multi_version_start=1713895401680243000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.449437] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861501][BatchIO][T0][Y0-0000000000000000-0-0] [lt=106] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:47.449586] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861504][BatchIO][T0][Y0-0000000000000000-0-0] [lt=90] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:47.449746] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861503][BatchIO][T0][Y0-0000000000000000-0-0] [lt=77] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:47.449948] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201067}, multi_version_start=1713895401680243000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.450082] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861513][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=151] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:47.450524] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=104] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201341}, multi_version_start=1713895401680243000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.451165] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200935}, multi_version_start=1713895401903950000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.451660] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201099}, multi_version_start=1713895401903950000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.451960] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [862263][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=167] ====== check clog disk timer task ====== [2024-04-24 10:00:47.452070] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [862263][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=93] get_disk_usage(ret=0, capacity(MB):=5529, used(MB):=1369) [2024-04-24 10:00:47.452286] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200409}, multi_version_start=1713895401903950000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.452724] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [862276][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=104] start do ls ha handler(ls_id_array_=[{id:1}, {id:1002}, {id:1001}]) [2024-04-24 10:00:47.452790] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200573}, multi_version_start=1713895402023024000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.453303] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=170] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200661}, multi_version_start=1713895402111532000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.453738] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201369}, multi_version_start=1713895402111532000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.454341] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201399}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.454860] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200141}, multi_version_start=1713895402327662000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.455453] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=183] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200259}, multi_version_start=1713895402327662000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.456158] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201149}, multi_version_start=1713895402531433000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.456675] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200377}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.457214] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200879}, multi_version_start=1713895402570777000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.457743] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200485}, multi_version_start=1713895402570777000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.458257] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [874528][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=70] ====== traversal_flush timer task ====== [2024-04-24 10:00:47.458313] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200511}, multi_version_start=1713895402728365000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.458453] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=130][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.458685] INFO [MDS] flush (mds_table_mgr.cpp:195) [874528][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=83] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60379}, rec_scn:{val:1713895222371319002, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1713895222371319003, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1713895222371319005, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1713895222371319006, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1713895222873479001, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1713895222873479002, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1713895222873479003, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1713895222873479005, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1713895222873479006, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1713895223156376001, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1713895223257209001, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1713895223257209002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1713895223257209003, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1713895223359838001, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1713895223359838002, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1713895223359838003, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1713895223460804001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1713895223460804002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1713895223562595001, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1713895223562595002, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1713895223562595004, v:0}}, {tablet_id:{id:378}, rec_scn:{val:1713895223562595005, v:0}}, {tablet_id:{id:101036}, rec_scn:{val:1713895223562595006, v:0}}, {tablet_id:{id:60005}, rec_scn:{val:1713895223562595007, v:0}}, {tablet_id:{id:60130}, rec_scn:{val:1713895223562595008, v:0}}, {tablet_id:{id:101064}, rec_scn:{val:1713895223562595010, v:0}}, {tablet_id:{id:101080}, rec_scn:{val:1713895223562595011, v:0}}, {tablet_id:{id:50424}, rec_scn:{val:1713895223562595012, v:0}}, {tablet_id:{id:197}, rec_scn:{val:1713895223562595013, v:0}}, {tablet_id:{id:50381}, rec_scn:{val:1713895223877482001, v:0}}, {tablet_id:{id:103}, rec_scn:{val:1713895223877482003, v:0}}, {tablet_id:{id:50373}, rec_scn:{val:1713895223877482004, v:0}}, }, max_consequent_callbacked_scn={val:1713924046506776000, v:0}, *this={this:0x14b0efaf8510, is_inited:true, freezing_scn:{val:1713879137201785000, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895205050766000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924042488319001, v:0}, tablet_change_checkpoint_scn:{val:1713877445609286000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924058572929000, latest_log_ts:{val:1713924041908950001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713919200442801001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:5, palf_env_:0x14b176ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:47.458844] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200585}, multi_version_start=1713895402728365000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.459094] INFO [MDS] flush (mds_table_mgr.cpp:195) [874528][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=278] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:9, ordered_tablets:{{tablet_id:{id:1152921504606847570}, rec_scn:{val:1713924005270741002, v:0}}, {tablet_id:{id:1152921504606847571}, rec_scn:{val:1713924009981239003, v:0}}, {tablet_id:{id:1152921504606847572}, rec_scn:{val:1713924014707803001, v:0}}, {tablet_id:{id:200315}, rec_scn:{val:1713924014707803001, v:0}}, {tablet_id:{id:1152921504606847573}, rec_scn:{val:1713924016371113002, v:0}}, {tablet_id:{id:1152921504606847574}, rec_scn:{val:1713924018036186001, v:0}}, {tablet_id:{id:1152921504606847577}, rec_scn:{val:1713924032690543001, v:0}}, {tablet_id:{id:1152921504606847578}, rec_scn:{val:1713924034513325001, v:0}}, {tablet_id:{id:1152921504606847580}, rec_scn:{val:1713924036970344001, v:0}}, }, max_consequent_callbacked_scn={val:1713924046122585000, v:0}, *this={this:0x14b0c0f06510, is_inited:true, freezing_scn:{val:1713879643147421001, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1713895202258988000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924042488319001, v:0}, tablet_change_checkpoint_scn:{val:1713923988969639000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:7, palf_env_:0x14b176ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:2, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:47.459357] INFO [MDS] flush (mds_table_mgr.cpp:195) [874528][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=176] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:8, ordered_tablets:{{tablet_id:{id:1152921504606847566}, rec_scn:{val:1713923993909177001, v:0}}, {tablet_id:{id:1152921504606847567}, rec_scn:{val:1713924003752491002, v:0}}, {tablet_id:{id:1152921504606847568}, rec_scn:{val:1713924004450773001, v:0}}, {tablet_id:{id:1152921504606847569}, rec_scn:{val:1713924004958544001, v:0}}, {tablet_id:{id:1152921504606847575}, rec_scn:{val:1713924027071226001, v:0}}, {tablet_id:{id:1152921504606847576}, rec_scn:{val:1713924029579575001, v:0}}, {tablet_id:{id:200316}, rec_scn:{val:1713924034106200001, v:0}}, {tablet_id:{id:1152921504606847579}, rec_scn:{val:1713924035882919002, v:0}}, }, max_consequent_callbacked_scn={val:1713924046083050001, v:0}, *this={this:0x14b0c0eb6510, is_inited:true, freezing_scn:{val:1713923772239168001, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1002}, ls_create_status:1, clog_checkpoint_scn:{val:1713923772377160001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924043670180000, v:0}, tablet_change_checkpoint_scn:{val:1713923978662610000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:2, proposal_id:0, palf_env_:0x14b176ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:2, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:47.459441] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=119] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200871}, multi_version_start=1713895402852184001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.459512] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [874528][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=141] succeed to traversal_flush(ret=0, ls_cnt=3) [2024-04-24 10:00:47.460018] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201027}, multi_version_start=1713895402852184001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.460572] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200987}, multi_version_start=1713895402852184001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.461071] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201441}, multi_version_start=1713895402988388000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.461553] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200831}, multi_version_start=1713895402988388000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.462084] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200603}, multi_version_start=1713895402988388000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.462566] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200843}, multi_version_start=1713895403179617000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.463122] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201253}, multi_version_start=1713895403208015001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.463609] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201217}, multi_version_start=1713895403208015001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.464143] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201405}, multi_version_start=1713895403358624000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.464672] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200209}, multi_version_start=1713895403461765000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.465225] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=104] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201015}, multi_version_start=1713895403461765000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.465480] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=126] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.465616] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=126] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.465797] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200773}, multi_version_start=1713895403559676001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.466293] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200613}, multi_version_start=1713895403559676001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.466814] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200069}, multi_version_start=1713895403640472000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.467331] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200229}, multi_version_start=1713895404019446000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.467957] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200917}, multi_version_start=1713895404019446000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.468377] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201395}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.468947] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200359}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.469458] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201065}, multi_version_start=1713895405346408002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.470109] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201247}, multi_version_start=1713895405346408002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.470587] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200695}, multi_version_start=1713895405346408002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.471226] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201223}, multi_version_start=1713895405559235001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.471614] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201259}, multi_version_start=1713895405621939000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.472090] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201233}, multi_version_start=1713895405621939000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.472527] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201159}, multi_version_start=1713895405738077001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.473125] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201011}, multi_version_start=1713895405832107000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.473566] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200499}, multi_version_start=1713895405832107000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.474185] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=123] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201119}, multi_version_start=1713895405978605000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.474699] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200241}, multi_version_start=1713895406039710000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.475293] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200749}, multi_version_start=1713895406039710000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.475832] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200851}, multi_version_start=1713895406188842000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.476392] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=122] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200189}, multi_version_start=1713895406188842001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.476911] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200177}, multi_version_start=1713895406188842001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.477436] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201133}, multi_version_start=1713895406188842001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.478066] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201283}, multi_version_start=1713895406384021000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.478590] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201333}, multi_version_start=1713895406440477000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.479278] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200777}, multi_version_start=1713895406440477000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.479792] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200655}, multi_version_start=1713895406440477000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.480374] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=153] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201475}, multi_version_start=1713895406440477000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.480614] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [861695][T1_ArcSender][T1][YB420A000003-000616C2A8B8F0E7-0-0] [lt=113][errcode=-4018] try free send task failed(ret=-4018) [2024-04-24 10:00:47.480934] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [862440][T1001_ArcSender][T1001][YB420A000003-000616C2AF28F0E7-0-0] [lt=70][errcode=-4018] try free send task failed(ret=-4018) [2024-04-24 10:00:47.480934] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200111}, multi_version_start=1713895406667156000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.481393] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200369}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.481817] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201363}, multi_version_start=1713895406758424000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.482296] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200837}, multi_version_start=1713895406758424000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.482725] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200064}, multi_version_start=1713895406826204001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.483203] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200775}, multi_version_start=1713895406826204001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.483713] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=120] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200827}, multi_version_start=1713895406826204001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.484198] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200957}, multi_version_start=1713895407115400001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.484736] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201291}, multi_version_start=1713895407115400001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.485367] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=271] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201171}, multi_version_start=1713895407159013001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.485954] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201079}, multi_version_start=1713895407159013001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.486506] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201203}, multi_version_start=1713895407313066000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.487125] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200937}, multi_version_start=1713895407418645001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.487591] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200373}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.488145] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201337}, multi_version_start=1713895407467247000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.488723] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200641}, multi_version_start=1713895407467247000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.489289] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200205}, multi_version_start=1713895407632385000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.489837] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200195}, multi_version_start=1713895407632385000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.490354] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200487}, multi_version_start=1713895408017921003, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.490829] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200010}, multi_version_start=1713895408017921003, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.491331] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201189}, multi_version_start=1713895408017921003, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.491803] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=50] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201237}, multi_version_start=1713895408659959001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.492269] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=132] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200291}, multi_version_start=1713895408659959001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.492443] INFO [SERVER] reload_tenant_task (ob_tenant_tablet_ttl_mgr.cpp:1297) [875471][T1004_TTLTablet][T1004][YB420A000003-000616C2BB99030F-0-0] [lt=78] finish reload tenant task(local_tenant_task_={tenant_id:1004, task_id:-1, is_usr_trigger:false, need_check:false, is_dirty:false, ttl_continue:true, rsp_time:-1, state:4, is_reused:true}, tenant_task={gmt_create:0, gmt_modified:0, tenant_id:18446744073709551615, table_id:18446744073709551615, tablet_id:18446744073709551615, task_id:0, task_start_time:0, task_update_time:0, trigger_type:0, status:19, ttl_del_cnt:0, max_version_del_cnt:0, scan_cnt:0, row_key:"", ret_code:"OB_SUCCESS"}, is_leader=true) [2024-04-24 10:00:47.492561] INFO [SERVER] check_schema_version (ob_tenant_tablet_ttl_mgr.cpp:1314) [875471][T1004_TTLTablet][T1004][YB420A000003-000616C2BB99030F-0-0] [lt=112] schema changed(ret=0, ret="OB_SUCCESS", local_schema_version=1713924035742592, schema_version=1713924044295272) [2024-04-24 10:00:47.492660] WDIAG [SERVER] mark_tenant_need_check (ob_tenant_tablet_ttl_mgr.cpp:288) [875471][T1004_TTLTablet][T1004][YB420A000003-000616C2BB99030F-0-0] [lt=88][errcode=-4016] unpected task id is found(ret=-4016, ret="OB_ERR_UNEXPECTED", local_tenant_task_={tenant_id:1004, task_id:-1, is_usr_trigger:false, need_check:false, is_dirty:false, ttl_continue:true, rsp_time:-1, state:4, is_reused:true}) [2024-04-24 10:00:47.492755] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201357}, multi_version_start=1713895408659959001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.493255] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200907}, multi_version_start=1713895408659959001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.493652] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201277}, multi_version_start=1713895408659959001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.494191] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200137}, multi_version_start=1713895408659959001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.494597] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201053}, multi_version_start=1713895408862810000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.495123] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200006}, multi_version_start=1713895409076643001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.495530] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201101}, multi_version_start=1713895409124379000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.496169] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200549}, multi_version_start=1713895409124379000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.496691] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200389}, multi_version_start=1713895409347340000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.497262] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200311}, multi_version_start=1713895409347340000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.497754] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200159}, multi_version_start=1713895409347340000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.498334] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200051}, multi_version_start=1713895409537095000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.498841] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201211}, multi_version_start=1713895409537095000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.499420] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=159] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201255}, multi_version_start=1713895409802831001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.499981] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200859}, multi_version_start=1713895409802831001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.500159] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=84] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.500272] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=95] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.500430] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=146] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.500540] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200267}, multi_version_start=1713895409802831001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.501160] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200453}, multi_version_start=1713895409958427000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.501643] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201389}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.502118] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=51] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200211}, multi_version_start=1713895410046144000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.502575] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=107] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200847}, multi_version_start=1713895410148309000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.503057] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=106] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201293}, multi_version_start=1713895410919443000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.503488] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200397}, multi_version_start=1713895410919443000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.504001] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [861558][T1_Occam][T1][YB420A000003-000616C2A92989A4-0-0] [lt=60] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:47.504009] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201285}, multi_version_start=1713895411213735000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.504531] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200157}, multi_version_start=1713895411359167000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.505047] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201075}, multi_version_start=1713895411359167000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.505519] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200923}, multi_version_start=1713895411419197001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.505817] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=101] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8890687488, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:47.506049] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200965}, multi_version_start=1713895411419197001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.506581] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200961}, multi_version_start=1713895411419197001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.507145] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201319}, multi_version_start=1713895411419197001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.507707] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200901}, multi_version_start=1713895411419197001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.508303] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201257}, multi_version_start=1713895412707851000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.508943] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:1152921504606846979}, multi_version_start=1713895412707851000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.509513] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201465}, multi_version_start=1713895412707851000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.510095] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201029}, multi_version_start=1713895412707851000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.510570] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200741}, multi_version_start=1713895412801270000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.511174] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=160] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201183}, multi_version_start=1713895412801270000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.511682] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201251}, multi_version_start=1713895412906426001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.512259] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200165}, multi_version_start=1713895412906426001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.512721] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200175}, multi_version_start=1713895412906426001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.513170] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200769}, multi_version_start=1713895413090843000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.513602] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201263}, multi_version_start=1713895413090843000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.514137] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=158] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200018}, multi_version_start=1713895413120305000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.514599] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201185}, multi_version_start=1713895413120305000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.515108] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:834) [874425][T1004_BalanceEx][T1004][YB420A000003-000616C2BB69A615-0-0] [lt=118] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2024-04-24 10:00:47.515168] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200521}, multi_version_start=1713895413120305000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.515211] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:171) [874425][T1004_BalanceEx][T1004][YB420A000003-000616C2BB69A615-0-0] [lt=79] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2024-04-24 10:00:47.515570] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200125}, multi_version_start=1713895413239795001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.516040] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200058}, multi_version_start=1713895413305582000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.516446] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201041}, multi_version_start=1713895413305582000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.517015] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200365}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.517551] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201355}, multi_version_start=1713895413422302001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.518119] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200115}, multi_version_start=1713895413554525000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.518560] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200303}, multi_version_start=1713895413554525000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.519137] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201139}, multi_version_start=1713895413554525000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.519533] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201195}, multi_version_start=1713895413641577000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.520106] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200129}, multi_version_start=1713895414041191000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.520521] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200263}, multi_version_start=1713895415152093001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.521102] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200399}, multi_version_start=1713895417319702000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.521509] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200941}, multi_version_start=1713895417626946000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.521928] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=50] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201037}, multi_version_start=1713895417626946000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.522385] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200873}, multi_version_start=1713895417626946000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.522803] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201427}, multi_version_start=1713895417852997001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.523252] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:1152921504606846988}, multi_version_start=1713895417852997001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.523649] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200803}, multi_version_start=1713895417998382001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.524095] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200008}, multi_version_start=1713895417998382001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.524596] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=123] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200571}, multi_version_start=1713895417998382001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.525087] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200005}, multi_version_start=1713895418269273000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.525388] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=62][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.525585] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201455}, multi_version_start=1713895418269273000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.526080] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200543}, multi_version_start=1713895418269273000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.526544] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200547}, multi_version_start=1713895418433676001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.527096] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201087}, multi_version_start=1713895418433676001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.527544] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200449}, multi_version_start=1713895418521465001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.528060] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200173}, multi_version_start=1713895418521465001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.528585] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200719}, multi_version_start=1713895418584744000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.529160] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201301}, multi_version_start=1713895418650410000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.529606] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200863}, multi_version_start=1713895418650410000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.530112] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=107] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201113}, multi_version_start=1713895418861295001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.530566] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201469}, multi_version_start=1713895418861295001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.531712] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=730] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200001}, multi_version_start=1713895418861295001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.532509] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=407] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200733}, multi_version_start=1713895418946741000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.532968] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200795}, multi_version_start=1713895419034202000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.533415] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200139}, multi_version_start=1713895419034202000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.533964] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200955}, multi_version_start=1713895419034202000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.534355] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200885}, multi_version_start=1713895419251171000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.534852] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200639}, multi_version_start=1713895419251171000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.535343] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=132] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200823}, multi_version_start=1713895419315606001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.535832] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200939}, multi_version_start=1713895419315606001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.536319] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=146] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200273}, multi_version_start=1713895419556205000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.536791] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200517}, multi_version_start=1713895419556205000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.537259] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200991}, multi_version_start=1713895419556205000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.537774] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200071}, multi_version_start=1713895419701193000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.538289] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201177}, multi_version_start=1713895419740084002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.538751] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200353}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.539288] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=149] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201437}, multi_version_start=1713895419863010000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.539803] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200135}, multi_version_start=1713895419909626001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.540508] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=201] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200563}, multi_version_start=1713895420057111000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.541088] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200757}, multi_version_start=1713895420137314000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.541720] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200245}, multi_version_start=1713895420137314000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.541729] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=79] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.542297] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200465}, multi_version_start=1713895420335142000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.542743] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201157}, multi_version_start=1713895420335142000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.543263] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201391}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.543362] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262A-0-0] [lt=95][errcode=-4023] tx is 2pc logging(this={this:0x14b0fbedf050, ref:3, trans_id:{txid:338711}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924076767546, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046792910], ctx_create_time:1713924046778243}{ls_id:{id:1}, session_id:3221795357, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046789276, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046791807000, v:0}, log_ts:{val:1713924046791807000, v:0}, lsn:{lsn:298513878}, submit_ts:1713924046792973}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:338711}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338711}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046791807000, v:0}, start_scn:{val:1713924046791807000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924046791807000, v:0} commit_version={val:1713924046791807000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338711} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=215183838 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046791807000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046792973, block_frozen_memtable:null}) [2024-04-24 10:00:47.543535] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262A-0-0] [lt=163][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fbedf050, ref:3, trans_id:{txid:338711}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924076767546, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046792910], ctx_create_time:1713924046778243}{ls_id:{id:1}, session_id:3221795357, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046789276, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046791807000, v:0}, log_ts:{val:1713924046791807000, v:0}, lsn:{lsn:298513878}, submit_ts:1713924046792973}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:338711}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338711}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046791807000, v:0}, start_scn:{val:1713924046791807000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924046791807000, v:0} commit_version={val:1713924046791807000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338711} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=215183838 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046791807000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046792973, block_frozen_memtable:null}) [2024-04-24 10:00:47.543675] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262A-0-0] [lt=129][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338711}) [2024-04-24 10:00:47.543699] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200012}, multi_version_start=1713895421107380000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.543761] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262A-0-0] [lt=80][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338711}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047538795, cluster_id:1713874979}, expire_ts:1713924076788074, commit_start_scn:{val:1713924046780604000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.543846] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262A-0-0] [lt=65][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338711}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047538795, cluster_id:1713874979}, expire_ts:1713924076788074, commit_start_scn:{val:1713924046780604000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.544262] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200701}, multi_version_start=1713895421107380000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.544741] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200445}, multi_version_start=1713895421107380000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.545258] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200133}, multi_version_start=1713895421244322000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.545670] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201421}, multi_version_start=1713895421244322000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.546181] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200697}, multi_version_start=1713895421244322000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.546579] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200050}, multi_version_start=1713895421244322000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.547044] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200313}, multi_version_start=1713895421462730000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.547568] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200349}, multi_version_start=1713895421462730001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.548203] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200973}, multi_version_start=1713895421462730001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.548717] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=126] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200269}, multi_version_start=1713895421462730001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.549366] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200123}, multi_version_start=1713895421462730001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.549969] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201003}, multi_version_start=1713895421486757000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.550486] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201227}, multi_version_start=1713895421754477001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.551066] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200673}, multi_version_start=1713895421754477001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.551577] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200587}, multi_version_start=1713895421754477001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.552131] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201269}, multi_version_start=1713895421754477001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.552392] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E95-0-0] [lt=57][errcode=-4023] tx is 2pc logging(this={this:0x14b168f2d710, ref:3, trans_id:{txid:877456}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924646597291, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046606334], ctx_create_time:1713924046597291}{ls_id:{id:1001}, session_id:3221852891, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046603230, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046605790000, v:0}, log_ts:{val:1713924046605790000, v:0}, lsn:{lsn:1002185585}, submit_ts:1713924046605790}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877456}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877456}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046605790000, v:0}, start_scn:{val:1713924046605790000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924046605790000, v:0} commit_version={val:1713924046605790000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877456} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=737308951 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046605790000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046605790, block_frozen_memtable:null}) [2024-04-24 10:00:47.552689] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200509}, multi_version_start=1713895422532608000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.552634] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E95-0-0] [lt=235][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b168f2d710, ref:3, trans_id:{txid:877456}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924646597291, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046606334], ctx_create_time:1713924046597291}{ls_id:{id:1001}, session_id:3221852891, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046603230, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046605790000, v:0}, log_ts:{val:1713924046605790000, v:0}, lsn:{lsn:1002185585}, submit_ts:1713924046605790}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877456}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877456}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046605790000, v:0}, start_scn:{val:1713924046605790000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924046605790000, v:0} commit_version={val:1713924046605790000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877456} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=737308951 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046605790000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046605790, block_frozen_memtable:null}) [2024-04-24 10:00:47.552781] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E95-0-0] [lt=141][errcode=-4023] commit fail(ret=-4023, coord={id:1001}, tx_id={txid:877456}) [2024-04-24 10:00:47.552904] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E95-0-0] [lt=73][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877456}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924047547673, cluster_id:1713874979}, expire_ts:1713924558320000, commit_start_scn:{val:1713924046594982000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:47.552998] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E95-0-0] [lt=78][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877456}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924047547673, cluster_id:1713874979}, expire_ts:1713924558320000, commit_start_scn:{val:1713924046594982000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:47.553215] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201017}, multi_version_start=1713895422532608000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.553757] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201105}, multi_version_start=1713895422532608000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.554255] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=157] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201473}, multi_version_start=1713895423232419001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.554800] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200945}, multi_version_start=1713895423354048002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.555405] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=148] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200475}, multi_version_start=1713895423354048002, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.556034] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201107}, multi_version_start=1713895424274351000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.556540] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200151}, multi_version_start=1713895424414118000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.557136] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200791}, multi_version_start=1713895424471910000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.557749] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=54] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200155}, multi_version_start=1713895424471910000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.558334] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201043}, multi_version_start=1713895425209665001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.558589] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=93][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.558758] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [862262][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=86] ====== traversal_flush timer task ====== [2024-04-24 10:00:47.558931] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201193}, multi_version_start=1713895425209665001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.559442] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200593}, multi_version_start=1713895425209665001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.559354] INFO [MDS] flush (mds_table_mgr.cpp:195) [862262][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=162] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60379}, rec_scn:{val:1713895310268028001, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1713895310268028003, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1713895310525506001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1713895310525506002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1713895310678201001, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1713895310678201002, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1713895310852482001, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1713895310852482002, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1713895310852482003, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1713895310852482004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1713895310992911001, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1713895310992911002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1713895311111630001, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1713895311260927001, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1713895311260927002, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1713895311260927003, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1713895311398568001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1713895311588474001, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1713895311588474003, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1713895311588474004, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1713895311783317001, v:0}}, {tablet_id:{id:378}, rec_scn:{val:1713895311783317002, v:0}}, {tablet_id:{id:101036}, rec_scn:{val:1713895311783317003, v:0}}, {tablet_id:{id:60005}, rec_scn:{val:1713895311783317004, v:0}}, {tablet_id:{id:60130}, rec_scn:{val:1713895311984459001, v:0}}, {tablet_id:{id:101064}, rec_scn:{val:1713895311984459002, v:0}}, {tablet_id:{id:101080}, rec_scn:{val:1713895311984459003, v:0}}, {tablet_id:{id:50424}, rec_scn:{val:1713895311984459004, v:0}}, {tablet_id:{id:197}, rec_scn:{val:1713895311984459005, v:0}}, {tablet_id:{id:50381}, rec_scn:{val:1713895312133482001, v:0}}, {tablet_id:{id:103}, rec_scn:{val:1713895312133482002, v:0}}, {tablet_id:{id:50373}, rec_scn:{val:1713895312274829001, v:0}}, }, max_consequent_callbacked_scn={val:1713924046061364000, v:0}, *this={this:0x14b147070510, is_inited:true, freezing_scn:{val:1713877162774754000, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895310268028001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924042914139000, v:0}, tablet_change_checkpoint_scn:{val:1713875507773116000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924056491232000, latest_log_ts:{val:1713924039886464001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713890395635598001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:5, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:47.560052] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=118] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200783}, multi_version_start=1713895425409972000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.560220] INFO [MDS] flush (mds_table_mgr.cpp:195) [862262][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=409] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:200589}, rec_scn:{val:1713895385940970001, v:0}}, {tablet_id:{id:201365}, rec_scn:{val:1713895386138343001, v:0}}, {tablet_id:{id:200929}, rec_scn:{val:1713895386138343002, v:0}}, {tablet_id:{id:200529}, rec_scn:{val:1713895386246298001, v:0}}, {tablet_id:{id:200197}, rec_scn:{val:1713895386246298003, v:0}}, {tablet_id:{id:200127}, rec_scn:{val:1713895386443363001, v:0}}, {tablet_id:{id:200251}, rec_scn:{val:1713895386443363003, v:0}}, {tablet_id:{id:201051}, rec_scn:{val:1713895386883002001, v:0}}, {tablet_id:{id:200767}, rec_scn:{val:1713895386883002002, v:0}}, {tablet_id:{id:201463}, rec_scn:{val:1713895387425291001, v:0}}, {tablet_id:{id:200999}, rec_scn:{val:1713895387572813001, v:0}}, {tablet_id:{id:201235}, rec_scn:{val:1713895388449421002, v:0}}, {tablet_id:{id:201309}, rec_scn:{val:1713895388449421004, v:0}}, {tablet_id:{id:200325}, rec_scn:{val:1713895388659263001, v:0}}, {tablet_id:{id:201025}, rec_scn:{val:1713895388717594001, v:0}}, {tablet_id:{id:200971}, rec_scn:{val:1713895388717594003, v:0}}, {tablet_id:{id:201081}, rec_scn:{val:1713895388717594005, v:0}}, {tablet_id:{id:200347}, rec_scn:{val:1713895389791968002, v:0}}, {tablet_id:{id:201151}, rec_scn:{val:1713895389992852001, v:0}}, {tablet_id:{id:200913}, rec_scn:{val:1713895389992852002, v:0}}, {tablet_id:{id:201483}, rec_scn:{val:1713895390122768001, v:0}}, {tablet_id:{id:201351}, rec_scn:{val:1713895390122768002, v:0}}, {tablet_id:{id:201205}, rec_scn:{val:1713895390122768004, v:0}}, {tablet_id:{id:201335}, rec_scn:{val:1713895390348851001, v:0}}, {tablet_id:{id:200911}, rec_scn:{val:1713895390455961001, v:0}}, {tablet_id:{id:200193}, rec_scn:{val:1713895390455961003, v:0}}, {tablet_id:{id:201239}, rec_scn:{val:1713895390714357007, v:0}}, {tablet_id:{id:200301}, rec_scn:{val:1713895391010940001, v:0}}, {tablet_id:{id:200703}, rec_scn:{val:1713895391146241001, v:0}}, {tablet_id:{id:200379}, rec_scn:{val:1713895391146241002, v:0}}, {tablet_id:{id:201375}, rec_scn:{val:1713895391240940001, v:0}}, {tablet_id:{id:201449}, rec_scn:{val:1713895391319921001, v:0}}, }, max_consequent_callbacked_scn={val:1713924046061364000, v:0}, *this={this:0x14b13438c510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1713895385940970001, v:0}, clog_base_lsn:{lsn:335523840}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924044057788000, v:0}, tablet_change_checkpoint_scn:{val:1713890381649962000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:7, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:47.560488] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200889}, multi_version_start=1713895425409972000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.561043] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200169}, multi_version_start=1713895425409972000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.560858] INFO [MDS] flush (mds_table_mgr.cpp:195) [862262][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=286] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:200876}, rec_scn:{val:1713895298278308001, v:0}}, {tablet_id:{id:200048}, rec_scn:{val:1713895298438136001, v:0}}, {tablet_id:{id:201384}, rec_scn:{val:1713895298438136002, v:0}}, {tablet_id:{id:200608}, rec_scn:{val:1713895298550042001, v:0}}, {tablet_id:{id:200450}, rec_scn:{val:1713895298611638001, v:0}}, {tablet_id:{id:200810}, rec_scn:{val:1713895298611638002, v:0}}, {tablet_id:{id:201394}, rec_scn:{val:1713895298744317001, v:0}}, {tablet_id:{id:201250}, rec_scn:{val:1713895298744317002, v:0}}, {tablet_id:{id:201318}, rec_scn:{val:1713895298744317003, v:0}}, {tablet_id:{id:200822}, rec_scn:{val:1713895298900066001, v:0}}, {tablet_id:{id:200874}, rec_scn:{val:1713895298900066002, v:0}}, {tablet_id:{id:200880}, rec_scn:{val:1713895299047127001, v:0}}, {tablet_id:{id:201270}, rec_scn:{val:1713895299149533001, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1713895299149533002, v:0}}, {tablet_id:{id:201434}, rec_scn:{val:1713895299252327001, v:0}}, {tablet_id:{id:201126}, rec_scn:{val:1713895299252327002, v:0}}, {tablet_id:{id:200124}, rec_scn:{val:1713895299252327003, v:0}}, {tablet_id:{id:200976}, rec_scn:{val:1713895299252327004, v:0}}, {tablet_id:{id:200566}, rec_scn:{val:1713895299446962001, v:0}}, {tablet_id:{id:201450}, rec_scn:{val:1713895299446962003, v:0}}, {tablet_id:{id:200312}, rec_scn:{val:1713895299446962004, v:0}}, {tablet_id:{id:200690}, rec_scn:{val:1713895299446962005, v:0}}, {tablet_id:{id:200808}, rec_scn:{val:1713895299654655001, v:0}}, {tablet_id:{id:200130}, rec_scn:{val:1713895299654655002, v:0}}, {tablet_id:{id:201138}, rec_scn:{val:1713895299654655003, v:0}}, {tablet_id:{id:200314}, rec_scn:{val:1713895299654655004, v:0}}, {tablet_id:{id:200031}, rec_scn:{val:1713895299654655005, v:0}}, {tablet_id:{id:200662}, rec_scn:{val:1713895299850465001, v:0}}, {tablet_id:{id:200492}, rec_scn:{val:1713895299955311001, v:0}}, {tablet_id:{id:200678}, rec_scn:{val:1713895299955311002, v:0}}, {tablet_id:{id:200332}, rec_scn:{val:1713895300142160001, v:0}}, {tablet_id:{id:201114}, rec_scn:{val:1713895300142160002, v:0}}, }, max_consequent_callbacked_scn={val:1713924046174250000, v:0}, *this={this:0x14b133736510, is_inited:true, freezing_scn:{val:1713890363233517001, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_create_status:1, clog_checkpoint_scn:{val:1713895298278308001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924044057788000, v:0}, tablet_change_checkpoint_scn:{val:1713890384892147000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:2, proposal_id:0, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:1942234752, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:47.561235] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [862262][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=347] succeed to traversal_flush(ret=0, ls_cnt=3) [2024-04-24 10:00:47.561562] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201457}, multi_version_start=1713895425409972000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.562095] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201347}, multi_version_start=1713895426387934000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.562489] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [862259][T1002_ReqMemEvi][T1002][Y0-0000000000000000-0-0] [lt=72] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=16106127, evict_high_size_level=90000, evict_batch_count=0, elapse_time=10, size_used=6588, mem_used=12478464) [2024-04-24 10:00:47.562591] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200525}, multi_version_start=1713895426450121001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.563161] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200061}, multi_version_start=1713895426450121001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.563580] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201103}, multi_version_start=1713895426450121001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.564133] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200555}, multi_version_start=1713895426631229000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.564585] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201299}, multi_version_start=1713895426664816000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.565111] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201377}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.565614] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200793}, multi_version_start=1713895426664816000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.565662] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=126] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.565753] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=85] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.566136] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201129}, multi_version_start=1713895426664816000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.566569] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200479}, multi_version_start=1713895426859091000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.567118] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200367}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.567614] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200609}, multi_version_start=1713895426859091000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.568191] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200014}, multi_version_start=1713895427021174000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.568643] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201013}, multi_version_start=1713895427107947001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.569212] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201061}, multi_version_start=1713895427107947001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.569664] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200343}, multi_version_start=1713895427213192000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.570190] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200763}, multi_version_start=1713895427213192000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.570657] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200833}, multi_version_start=1713895427283357000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.571274] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201023}, multi_version_start=1713895427445256000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.571858] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200591}, multi_version_start=1713895427473102000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.572531] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=160] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201381}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.573241] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=119] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200333}, multi_version_start=1713895427555666000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.573961] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=215] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200575}, multi_version_start=1713895427555666000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.574192] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [862430][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=116] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=90000, evict_batch_count=0, elapse_time=6, size_used=8397, mem_used=14558208) [2024-04-24 10:00:47.574475] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201077}, multi_version_start=1713895427555666000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.575086] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200681}, multi_version_start=1713895427555666000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.575564] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201461}, multi_version_start=1713895427669993001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.576084] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200817}, multi_version_start=1713895427856654000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.576520] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200277}, multi_version_start=1713895427893561000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.577011] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200257}, multi_version_start=1713895427893561000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.577446] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200581}, multi_version_start=1713895428069550001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.578024] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=107] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200981}, multi_version_start=1713895428069550001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.578460] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200969}, multi_version_start=1713895428069550001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.579078] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=136] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201267}, multi_version_start=1713895428298606000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.579691] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200463}, multi_version_start=1713895428298606000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.580398] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=140] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201007}, multi_version_start=1713895428298606000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.581013] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201163}, multi_version_start=1713895428438419000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.581503] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200455}, multi_version_start=1713895428500114000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.582079] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200693}, multi_version_start=1713895428500114000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.582562] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200943}, multi_version_start=1713895428653828001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.583142] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201453}, multi_version_start=1713895428653828001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.583669] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200481}, multi_version_start=1713895428712043000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.584180] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201383}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.584668] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201371}, multi_version_start=1713920416186806001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.585240] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200649}, multi_version_start=1713895428804584000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.585695] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] REACH SYSLOG RATE LIMIT [frequency] [2024-04-24 10:00:47.589246] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [861688][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=149] ====== traversal_flush timer task ====== [2024-04-24 10:00:47.589690] INFO [MDS] flush (mds_table_mgr.cpp:195) [861688][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=109] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1713895261489897001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1713895262106069001, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1713895262106069002, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1713895262608522001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1713895262608522002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1713895263027701001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1713895263027701002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1713895263027701003, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1713895263109231001, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1713895263201088001, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1713895263226856002, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1713895263471989001, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1713895263515510001, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1713895263616210002, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1713895263616210003, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1713895263734257001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1713895263795311001, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1713895263838851001, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1713895263891959001, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1713895263936540001, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1713895263945526001, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1713895263990416001, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1713895264026604001, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1713895264026604002, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1713895264026604003, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1713895264026604004, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1713895264138915001, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1713895264138915002, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1713895264138915003, v:0}}, {tablet_id:{id:60342}, rec_scn:{val:1713895264138915004, v:0}}, {tablet_id:{id:378}, rec_scn:{val:1713895264138915005, v:0}}, {tablet_id:{id:101036}, rec_scn:{val:1713895264138915006, v:0}}, }, max_consequent_callbacked_scn={val:1713924046191046000, v:0}, *this={this:0x14b186b16510, is_inited:true, freezing_scn:{val:1713895202277178000, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895204295552002, v:0}, clog_base_lsn:{lsn:67104768}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1713875366171467000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924058636985000, latest_log_ts:{val:1713924041985593001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713918035834957001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1713903554859671002, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:2, proposal_id:0, palf_env_:0x14b1945f8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:47.589859] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [861688][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=141] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-04-24 10:00:47.593353] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2608-0-0] [lt=64][errcode=-4023] tx is 2pc logging(this={this:0x14b15a16b090, ref:3, trans_id:{txid:391371}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924076773625, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046800195], ctx_create_time:1713924046784446}{ls_id:{id:1}, session_id:3221795340, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046791808, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046799990000, v:0}, log_ts:{val:1713924046799990000, v:0}, lsn:{lsn:314127056}, submit_ts:1713924046799990}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:391371}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:391371}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046799990000, v:0}, start_scn:{val:1713924046799990000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875442714112 max_table_version=1713875442714112 trans_version={val:1713924046799990000, v:0} commit_version={val:1713924046799990000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:391371} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=2612980561 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046799990000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046799990, block_frozen_memtable:null}) [2024-04-24 10:00:47.593538] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2608-0-0] [lt=171][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b15a16b090, ref:3, trans_id:{txid:391371}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924076773625, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046800195], ctx_create_time:1713924046784446}{ls_id:{id:1}, session_id:3221795340, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046791808, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046799990000, v:0}, log_ts:{val:1713924046799990000, v:0}, lsn:{lsn:314127056}, submit_ts:1713924046799990}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:391371}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:391371}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046799990000, v:0}, start_scn:{val:1713924046799990000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875442714112 max_table_version=1713875442714112 trans_version={val:1713924046799990000, v:0} commit_version={val:1713924046799990000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:391371} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=2612980561 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046799990000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046799990, block_frozen_memtable:null}) [2024-04-24 10:00:47.593722] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2608-0-0] [lt=170][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:391371}) [2024-04-24 10:00:47.593819] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2608-0-0] [lt=90][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1001, tx_id:{txid:391371}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047588882, cluster_id:1713874979}, expire_ts:1713924076795547, commit_start_scn:{val:1713924046786976000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.593945] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2608-0-0] [lt=106][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1001, tx_id:{txid:391371}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047588882, cluster_id:1713874979}, expire_ts:1713924076795547, commit_start_scn:{val:1713924046786976000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.600308] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=77] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.600424] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=97] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.600522] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=82] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.604401] INFO [SQL.JO] compute_table_location (ob_join_order.cpp:278) [861734][T1_L0_G0][T1][YB420A000003-000616C2ADE8FCD2-0-0] [lt=82] succeed to calculate base table sharding info(table_id=18446744073709551614, ref_table_id=104, is_global_index=false) [2024-04-24 10:00:47.604525] INFO [SQL.JO] init_est_sel_info_for_access_path (ob_join_order.cpp:12189) [861734][T1_L0_G0][T1][YB420A000003-000616C2ADE8FCD2-0-0] [lt=97] total rowcount, use statistics(table_meta_info_.table_row_count_=10, table_meta_info_.average_row_size_=2.060000000000000000e+02, table_meta_info_.micro_block_count_=1, table_meta_info_.part_size_=2.060000000000000000e+03) [2024-04-24 10:00:47.605040] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [861686][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=81] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-24 10:00:47.605047] WDIAG [STORAGE.TRANS] inner_lock_for_read (ob_tx_data_functor.cpp:290) [879578][T1003_L0_G0][T1003][YB420A000002-000616C2B7CC236D-0-0] [lt=89][errcode=-6004] lock_for_read need retry(ret=-6004, tx_data={tx_id:{txid:548715}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046506776000, v:0}, start_scn:{val:1713924046506776000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}, lock_for_read_arg_={mvcc_acc_ctx_:{type:1, abs_lock_timeout:1713924076814002, tx_lock_timeout:-1, snapshot:{version:{val:1713924046909965000, v:0}, tx_id:{txid:548718}, scn:1713924046913660}, tx_table_guards:{tx_table_guard:{tx_table:0x14b0f0a76490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, data_trans_id_:{txid:548715}, data_sql_sequence_:1713924046505649, read_latest_:false, scn_:{val:4611686018427387903, v:0}}, tx_cc_ctx={state:10, prepare_version:{val:1713924046506776000, v:0}}) [2024-04-24 10:00:47.614695] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [862135][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=138] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}]) [2024-04-24 10:00:47.619544] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [874532][T1004_LSArchive][T1004][YB420A000003-000616C2B568F0E7-0-0] [lt=67] gc stale ls task succ [2024-04-24 10:00:47.625647] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=66][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.628069] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [861673][T1_LSMetaCh][T1][YB420A000003-000616C2A909ADA1-0-0] [lt=122] fetch root ls success(local_cluster_id=1713874979, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.2:2882", sql_port:2881, role:1, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:5, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713921656566585, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.3:2882", sql_port:2881, role:2, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["10.0.0.2:2882", "10.0.0.3:2882"]) [2024-04-24 10:00:47.633539] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=70] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-04-24 10:00:47.637139] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200849}, multi_version_start=1713895443514046001, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.643715] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [861685][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=69] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=8, size_used=15899, mem_used=27036672) [2024-04-24 10:00:47.642734] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=100] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.649833] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AA-0-0] [lt=78][errcode=-4023] tx is 2pc logging(this={this:0x14b148003290, ref:4, trans_id:{txid:176905}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924076581982, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589598], ctx_create_time:1713924046582067}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046580909, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452000, v:0}, log_ts:{val:1713924046589452000, v:0}, lsn:{lsn:79580022}, submit_ts:1713924046589452}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176905}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176905}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452000, v:0}, start_scn:{val:1713924046589452000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465665976 max_table_version=1713877465684248 trans_version={val:1713924046589452000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176905} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=924633448 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:47.650095] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AA-0-0] [lt=246][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b148003290, ref:4, trans_id:{txid:176905}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924076581982, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589598], ctx_create_time:1713924046582067}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046580909, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452000, v:0}, log_ts:{val:1713924046589452000, v:0}, lsn:{lsn:79580022}, submit_ts:1713924046589452}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176905}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176905}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452000, v:0}, start_scn:{val:1713924046589452000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465665976 max_table_version=1713877465684248 trans_version={val:1713924046589452000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176905} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=924633448 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:47.650243] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AA-0-0] [lt=138][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176905}) [2024-04-24 10:00:47.650348] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AA-0-0] [lt=91][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176905}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924047639906, cluster_id:1713874979}, expire_ts:1713924076505566, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.650519] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AA-0-0] [lt=153][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176905}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924047639906, cluster_id:1713874979}, expire_ts:1713924076505566, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.650832] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A9-0-0] [lt=79][errcode=-4023] tx is 2pc logging(this={this:0x14b055e1a050, ref:3, trans_id:{txid:176904}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138442, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589701], ctx_create_time:1713924046585672}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046585674, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452001, v:0}, log_ts:{val:1713924046589452001, v:0}, lsn:{lsn:79580890}, submit_ts:1713924046589452}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176904}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176904}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452001, v:0}, start_scn:{val:1713924046589452001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877457407528 trans_version={val:1713924046589452001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176904} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3140423830 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:47.651079] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A9-0-0] [lt=239][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e1a050, ref:3, trans_id:{txid:176904}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138442, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589701], ctx_create_time:1713924046585672}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046585674, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452001, v:0}, log_ts:{val:1713924046589452001, v:0}, lsn:{lsn:79580890}, submit_ts:1713924046589452}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176904}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176904}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452001, v:0}, start_scn:{val:1713924046589452001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877457407528 trans_version={val:1713924046589452001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176904} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3140423830 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:47.651217] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [861734][T1_L0_G0][T1][YB420A000003-000616C2ADE8FCD2-0-0] [lt=79][errcode=0] set logical del time(cache_obj->get_logical_del_time()=576360225329, cache_obj->added_lc()=false, cache_obj->get_object_id()=39595, cache_obj->get_tenant_id()=1, lbt()="0x121f36ac 0xc327dd4 0x4e3fdbe 0x4e31e28 0x4e29b67 0x4e21201 0x4e1eb17 0x4e0e175 0xa7ac354 0x12d1fdbc 0x14b1c1d132de 0x14b1c1a44e83") [2024-04-24 10:00:47.651254] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A9-0-0] [lt=158][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176904}) [2024-04-24 10:00:47.651344] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A9-0-0] [lt=79][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176904}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924047639583, cluster_id:1713874979}, expire_ts:1713925045138403, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.651486] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04A9-0-0] [lt=119][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176904}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924047639583, cluster_id:1713874979}, expire_ts:1713925045138403, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.657192] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [862435][T1001_ObTimer][T1001][Y0-0000000000000000-0-0] [lt=153] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-04-24 10:00:47.657321] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [862435][T1001_ObTimer][T1001][Y0-0000000000000000-0-0] [lt=84] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-04-24 10:00:47.657428] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [862435][T1001_ObTimer][T1001][Y0-0000000000000000-0-0] [lt=94] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=9726) [2024-04-24 10:00:47.658778] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=143][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.665796] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=71] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.665910] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=108] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.671839] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [874706][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=131] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:769, free_obj_hold_cnt:74, allocator used:3351768, allocator total:3466624}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:72, free_obj_hold_cnt:32, allocator used:6821464, allocator total:6821464}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():154752, full allocator used:1600, full allocator total:220288}, tablets_mem=10508376, tablets_mem_limit=257698020, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:23, free_obj_hold_cnt:163, allocator used:369024, allocator total:392448}, pending_cnt=0, wait_gc_count=0, tablet count=841) [2024-04-24 10:00:47.690708] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=35] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:200561}, multi_version_start=1713895459442819000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.700528] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=82] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.700610] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=70] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.700682] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=66] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.706717] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=130] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8890687488, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:47.707354] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=109] ====== checkpoint timer task ====== [2024-04-24 10:00:47.707546] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=123] get_max_applied_scn(scn={val:1713924046050326000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:5, palf_committed_end_lsn_:{lsn:314125850}, last_check_scn_:{val:1713924046248201001, v:0}, max_applied_cb_scn_:{val:1713924046050326000, v:0}}) [2024-04-24 10:00:47.707694] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1018) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=135] get_min_unreplayed_log_info in skip state(lsn={lsn:299748699}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713921656836981, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:299748699}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:299748699}, end_lsn:{lsn:299748699}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x14b14c9b9470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-04-24 10:00:47.709056] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=149] succ to get rec scn(*this={this:0x14b14b804030, ls_id:{id:1}, tenant_id:1001, state:"L_WORKING", total_tx_ctx_count:5, active_tx_count:5, total_active_readonly_request_count:1, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1713895201571489000, v:0}, last_push_gc_task_ts_:1713895203315693, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:844}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-24 10:00:47.709144] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=82] tx ctx memtable get rec scn(this={ObITable:{this:0x14b14be62080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1713895203319956, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x14b14be62080, snapshot_version:{val:1713895203319956, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:1713924046221727000, v:0}) [2024-04-24 10:00:47.709254] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=76] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-04-24 10:00:47.709586] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=72] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1713895313405863001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1713895313405863001, v:0}}, }, *this={this:0x14b14c97a510, is_inited:true, freezing_scn:{val:1713877132047340000, v:0}, ls:{ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895203696037000, v:0}, clog_base_lsn:{lsn:134209536}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1713875407740689002, v:0}, all_id_meta:{id_meta:[{limited_id:1713924059338456000, latest_log_ts:{val:1713924042694950001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713898575276749001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1713910854505533001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:5, palf_env_:0x14b157ff2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:47.709823] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=222] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x14b14c988b90}, min_rec_scn={val:1713895203696037000, v:0}, ls_id_={id:1}) [2024-04-24 10:00:47.709955] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=119] get rec log scn(service_type_=0, rec_log_ts={val:1713924042694950001, v:0}) [2024-04-24 10:00:47.710038] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=75] get rec log scn(service_type_=1, rec_log_ts={val:1713898575276749001, v:0}) [2024-04-24 10:00:47.710121] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=75] get rec log scn(service_type_=2, rec_log_ts={val:1713910854505533001, v:0}) [2024-04-24 10:00:47.710206] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=78] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713895203696037000, v:0}, checkpoint_scn_in_ls_meta={val:1713895203696037000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-04-24 10:00:47.710285] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=64][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, curr_base_lsn={lsn:134209536}, new_base_lsn={lsn:134209536}, lsn={lsn:134209536}) [2024-04-24 10:00:47.710375] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=81] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:134209536}, ls->get_ls_id()={id:1}) [2024-04-24 10:00:47.710450] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=69] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-04-24 10:00:47.723296] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [861679][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=78] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=1, total_destructed_detector=1, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x14b1987ea2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-04-24 10:00:47.725773] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=88][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.733372] INFO [LIB] stat (utility.h:1197) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=86] [PALF STAT WRITE LOG COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1157172, this=0x14b13adcd3b0, extra_info="palf_id:1002") [2024-04-24 10:00:47.733496] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=116][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:165096706}], scn_array=[{val:1713924046271305000, v:0}], curr_size=789, accum_size=789, time_cost=1157172) [2024-04-24 10:00:47.733546] INFO [LIB] stat (utility.h:1197) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=146] [PALF STAT WRITE LOG COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1160466, this=0x14b0f0aab3b0, extra_info="palf_id:1") [2024-04-24 10:00:47.733573] INFO [LIB] stat (utility.h:1197) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=154] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=246333, this=0x14b14c9bd3b0, extra_info="palf_id:1") [2024-04-24 10:00:47.733604] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=98] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=789) [2024-04-24 10:00:47.733677] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=119][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:298512772}], scn_array=[{val:1713924046197958001, v:0}], curr_size=564, accum_size=564, time_cost=1160466) [2024-04-24 10:00:47.733708] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=122][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:314125850}], scn_array=[{val:1713924046221727000, v:0}], curr_size=1084, accum_size=4147, time_cost=1159591) [2024-04-24 10:00:47.733754] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=71] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=564) [2024-04-24 10:00:47.733774] INFO [LIB] stat (utility.h:1214) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=125] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=142, this=0x14b1679f73b0, extra_info="") [2024-04-24 10:00:47.733774] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=59] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=4147) [2024-04-24 10:00:47.733896] INFO [LIB] stat (utility.h:1197) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT LOG IO WORKER QUEUE SIZE](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=1, this=0x14b1679f7068, extra_info="log_io_worker_num:4, cb_thread_pool_tg_id:395") [2024-04-24 10:00:47.733902] INFO [LIB] stat (utility.h:1214) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=62] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=562, this=0x14b1421a2530, extra_info="") [2024-04-24 10:00:47.733928] INFO [LIB] stat (utility.h:1214) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=88] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=335, this=0x14b157ff6530, extra_info="") [2024-04-24 10:00:47.734096] INFO [LIB] stat (utility.h:1197) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=87] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=289319, this=0x14b1336f53b0, extra_info="palf_id:1001") [2024-04-24 10:00:47.734227] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=281][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="10.0.0.3:2882", server="10.0.0.3:2882", update_func={old_end_lsn:{lsn:298512772}, new_end_lsn:{lsn:298513336}, old_advance_time_us:1713924046572261, new_ack_time_us:1713924047734142, advance delay(us):1161881}) [2024-04-24 10:00:47.734284] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=145][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:1002185407}], scn_array=[{val:1713924046189257001, v:0}], curr_size=122, accum_size=7986, time_cost=1158811) [2024-04-24 10:00:47.734357] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=122] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=1761) [2024-04-24 10:00:47.734371] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=80] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=7986) [2024-04-24 10:00:47.734440] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=118] [PALF STAT FLUSH CB COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=107, this=0x14b13adcd5e0, extra_info="palf_id:1002") [2024-04-24 10:00:47.734469] INFO [LIB] stat (utility.h:1214) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=69] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=500, this=0x14b1679f6c70, extra_info="") [2024-04-24 10:00:47.734537] INFO [LIB] stat (utility.h:1197) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=67] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=124, this=0x14b0f0aa1470, extra_info="") [2024-04-24 10:00:47.734569] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=68][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1001, self="10.0.0.3:2882", server="10.0.0.3:2882", update_func={old_end_lsn:{lsn:1002185407}, new_end_lsn:{lsn:1002185529}, old_advance_time_us:1713924046574703, new_ack_time_us:1713924047734562, advance delay(us):1159859}) [2024-04-24 10:00:47.734669] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=63] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=4, avg_log_batch_cnt=1, total_group_log_size=3260, avg_group_log_size=815) [2024-04-24 10:00:47.734669] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=86] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="10.0.0.3:2882", committed size=122) [2024-04-24 10:00:47.734669] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=90][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="10.0.0.3:2882", server="10.0.0.3:2882", update_func={old_end_lsn:{lsn:314125850}, new_end_lsn:{lsn:314126934}, old_advance_time_us:1713924046573528, new_ack_time_us:1713924047734655, advance delay(us):1161127}) [2024-04-24 10:00:47.734786] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=105] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=5162) [2024-04-24 10:00:47.734819] INFO [LIB] stat (utility.h:1197) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=75] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=39, this=0x14b0f0aa16a0, extra_info="palf_id:1") [2024-04-24 10:00:47.734968] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=84, this=0x14b1336eb470, extra_info="") [2024-04-24 10:00:47.734990] INFO [LIB] stat (utility.h:1197) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=149] [PALF STAT LOG LIFE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=784684, this=0x14b0f0aa18d0, extra_info="palf_id:1") [2024-04-24 10:00:47.735068] INFO [LIB] stat (utility.h:1197) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=54] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=90, this=0x14b14c9b3470, extra_info="") [2024-04-24 10:00:47.735083] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=85] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=3, avg_log_gen_to_freeze_time=77142, avg_log_gen_to_submit_time=77337, avg_log_submit_to_flush_time=515784, avg_log_submit_to_slide_time=707347) [2024-04-24 10:00:47.735090] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=92] [PALF STAT GROUP LOG INFO](palf_id=1001, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=7, avg_log_batch_cnt=1, total_group_log_size=9559, avg_group_log_size=1365) [2024-04-24 10:00:47.735159] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=55] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=7, avg_log_batch_cnt=1, total_group_log_size=4978, avg_group_log_size=711) [2024-04-24 10:00:47.735159] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=67][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=482542, log_task={header:{begin_lsn:{lsn:298512772}, end_lsn:{lsn:298513336}, log_id:482542, min_scn:{val:1713924046197958001, v:0}, max_scn:{val:1713924046506776000, v:0}, data_len:508, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:298511688}, data_checksum:2569666292, accum_checksum:594688838, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924046348696, freeze_ts:1713924046572267, submit_ts:1713924046572477, flushed_ts:1713924047734134, gen_to_freeze cost time:223571, gen_to_submit cost time:223781, submit_to_flush cost time:1161657}, fs_cb_begin_ts=1713924047734774, log_life_time=1386078) [2024-04-24 10:00:47.735208] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=69] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=27, this=0x14b1336eb6a0, extra_info="palf_id:1001") [2024-04-24 10:00:47.735265] INFO [LIB] stat (utility.h:1197) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=72] [PALF STAT FLUSH CB COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1119, this=0x14b0f0aab5e0, extra_info="palf_id:1") [2024-04-24 10:00:47.735285] INFO [LIB] stat (utility.h:1197) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=72] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=28, this=0x14b14c9b36a0, extra_info="palf_id:1") [2024-04-24 10:00:47.735292] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=70] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=399352, this=0x14b1336eb8d0, extra_info="palf_id:1001") [2024-04-24 10:00:47.735292] INFO [LIB] stat (utility.h:1197) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=156] [APPLY STAT CB APPEND COST TIME](cur_stat_count=50, stat_interval=5000000, avg (count/cost)=171, this=0x14b133646280, extra_info="") [2024-04-24 10:00:47.735360] INFO [LIB] stat (utility.h:1197) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=62] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=350784, this=0x14b14c9b38d0, extra_info="palf_id:1") [2024-04-24 10:00:47.735375] INFO [LIB] stat (utility.h:1197) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=68] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=50, stat_interval=5000000, avg (count/cost)=74133, this=0x14b1336464b0, extra_info="") [2024-04-24 10:00:47.735375] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT LOG TASK TIME](palf_id=1001, self="10.0.0.3:2882", total_slide_log_cnt=7, avg_log_gen_to_freeze_time=77912, avg_log_gen_to_submit_time=78098, avg_log_submit_to_flush_time=276065, avg_log_submit_to_slide_time=321254) [2024-04-24 10:00:47.735444] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=75] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=7, avg_log_gen_to_freeze_time=71304, avg_log_gen_to_submit_time=71454, avg_log_submit_to_flush_time=247108, avg_log_submit_to_slide_time=279330) [2024-04-24 10:00:47.735466] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=82][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=673413, log_task={header:{begin_lsn:{lsn:1002185407}, end_lsn:{lsn:1002185529}, log_id:673413, min_scn:{val:1713924046189257001, v:0}, max_scn:{val:1713924046189257001, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1002183834}, data_checksum:2222604892, accum_checksum:183913432, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924046291309, freeze_ts:1713924046574732, submit_ts:1713924046574855, flushed_ts:1713924047734556, gen_to_freeze cost time:283423, gen_to_submit cost time:283546, submit_to_flush cost time:1159701}, fs_cb_begin_ts=1713924047735172, log_life_time=1443863) [2024-04-24 10:00:47.735444] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=75][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924046248201000, v:0}, log_ts:{val:1713924046248201000, v:0}, lsn:{lsn:314126392}, submit_ts:1713924046248201}, this:0x14b15a167280, is_inited_:true, trans_id:{txid:568558}, ls_id:{id:1}, ctx:0x14b15a164310, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1487227, ctx_lock_wait_time=7) [2024-04-24 10:00:47.735466] INFO [LIB] stat (utility.h:1197) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=76] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=50, stat_interval=5000000, avg (count/cost)=0, this=0x14b1336466e0, extra_info="") [2024-04-24 10:00:47.735514] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=63][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=505454, log_task={header:{begin_lsn:{lsn:314125850}, end_lsn:{lsn:314126934}, log_id:505454, min_scn:{val:1713924046221727000, v:0}, max_scn:{val:1713924046248201001, v:0}, data_len:1028, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:314125728}, data_checksum:2643252445, accum_checksum:2629461264, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924046222185, freeze_ts:1713924046573544, submit_ts:1713924046573661, flushed_ts:1713924047734645, gen_to_freeze cost time:351359, gen_to_submit cost time:351476, submit_to_flush cost time:1160984}, fs_cb_begin_ts=1713924047735246, log_life_time=1513061) [2024-04-24 10:00:47.735572] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=65] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=331, this=0x14b1336f55e0, extra_info="palf_id:1001") [2024-04-24 10:00:47.735578] INFO [LIB] stat (utility.h:1197) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=100] [APPLY STAT CB EXECUTE TIME](cur_stat_count=50, stat_interval=5000000, avg (count/cost)=-496, this=0x14b133646910, extra_info="") [2024-04-24 10:00:47.735702] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=101][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924046506776000, v:0}, log_ts:{val:1713924046506776000, v:0}, lsn:{lsn:298512894}, submit_ts:1713924046506776}, this:0x14b0fbfc3d40, is_inited_:true, trans_id:{txid:548715}, ls_id:{id:1}, ctx:0x14b0fbfc0dd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1228881, ctx_lock_wait_time=14) [2024-04-24 10:00:47.735722] INFO [LIB] stat (utility.h:1197) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=131] [APPLY STAT CB TOTAL TIME](cur_stat_count=50, stat_interval=5000000, avg (count/cost)=73809, this=0x14b133646b40, extra_info="") [2024-04-24 10:00:47.735722] INFO [LIB] stat (utility.h:1197) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=179] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=327, this=0x14b14c9bd5e0, extra_info="palf_id:1") [2024-04-24 10:00:47.735802] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=87][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924046221727000, v:0}, log_ts:{val:1713924046221727000, v:0}, lsn:{lsn:314125906}, submit_ts:1713924046221727}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568557}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1514053, ctx_lock_wait_time=13) [2024-04-24 10:00:47.735814] WDIAG [CLOG] statistics_cb_cost_ (ob_log_apply_service.cpp:933) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=86][errcode=-4389] cb cost too much time(lsn={lsn:1002185463}, scn={val:1713924046189257001, v:0}, idx=9, total_cost_time=1441697, append_cost_time=114, cb_wait_thread_time=1443925, cb_wait_commit_time=0, cb_cost_time=-2342, append_start_time=1713924046291239, append_finish_time=1713924046291353, cb_first_handle_time=1713924047735278, cb_finish_time=1713924047732936) [2024-04-24 10:00:47.735828] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:188) [862397][T1001_FreInfoRe][T1001][Y0-0000000000000000-0-0] [lt=92] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1713922236767608000) [2024-04-24 10:00:47.735927] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [862260][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=94] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1713924045845177000, v:0}, new_max_version={val:1713924045845177000, v:0}, last_min_version={val:1713924045458795000, v:0}, last_max_version={val:1713924045458795000, v:0}, total_time_us=1489133, delta_us=1890678) [2024-04-24 10:00:47.736000] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1071) [862397][T1001_FreInfoRe][T1001][Y0-0000000000000000-0-0] [lt=157] update reserved snapshot finished(cost_ts=1477033, reserved_snapshot=1713922236767608000) [2024-04-24 10:00:47.736026] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [862260][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=93] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1713924045845177000, v:0}, min_version={val:1713924045845177000, v:0}, max_version={val:1713924045845177000, v:0}, in_service=true, leader_epoch=5) [2024-04-24 10:00:47.736134] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1001}, tablet_id={id:201005}, multi_version_start=1713895475824809000, min_reserved_snapshot=1713922230861204000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922230861204000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:47.736383] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [879578][T1003_L0_G0][T1003][YB420A000002-000616C2B7CC236D-0-0] [lt=112] current clock is (clock_=0, max_seq_=0, mem_limit=264912214, cur_mem_hold=121634816, attr_.tenant_id_=1003) [2024-04-24 10:00:47.736483] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [879578][T1003_L0_G0][T1003][YB420A000002-000616C2B7CC236D-0-0] [lt=79] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924036574474, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924047732936, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:47.736843] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [879578][T1003_L0_G0][T1003][YB420A000002-000616C2B7CC236D-0-0] [lt=160] get number(ret=0, service_type_=0, range=1, base_id=1713924047736671000, start_id=1713924047736671000, end_id=1713924047736671001) [2024-04-24 10:00:47.737046] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [879578][T1003_L0_G0][T1003][YB420A000002-000616C2B7CC236D-0-0] [lt=145] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=3373) [2024-04-24 10:00:47.737139] INFO [LIB] stat (utility.h:1197) [879578][T1003_L0_G0][T1003][YB420A000002-000616C2B7CC236D-0-0] [lt=65] [PALF STAT APPEND COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x14b0f0a74e50, extra_info="id:1") [2024-04-24 10:00:47.737559] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=184] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1713924045694942002, v:0}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=2042596, in_cluster_service=true, cluster_version={val:1713924045694942002, v:0}, min_cluster_version={val:1713924045694942002, v:0}, max_cluster_version={val:1713924045694942002, v:0}, get_cluster_version_err=0, cluster_version_delta=2042596, cluster_service_master="10.0.0.3:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=348127, succ_cluster_heartbeat_count=66, cluster_heartbeat_interval=50000, local_cluster_version={val:1713923391817005001, v:0}, local_cluster_delta=655920533, force_self_check=false, weak_read_refresh_interval=100000) [2024-04-24 10:00:47.737854] INFO do_work (ob_rl_mgr.cpp:709) [861413][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=162] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-24 10:00:47.738221] INFO [LIB] stat (utility.h:1197) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=62] [PALF STAT WRITE LOG COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1149346, this=0x14b186b513b0, extra_info="palf_id:1") [2024-04-24 10:00:47.738332] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=102][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179736232}], scn_array=[{val:1713924046351265000, v:0}], curr_size=1045, accum_size=1045, time_cost=1149346) [2024-04-24 10:00:47.738421] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=75] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=1045) [2024-04-24 10:00:47.738535] INFO [LIB] stat (utility.h:1214) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=81] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=129, this=0x14b1945fe530, extra_info="") [2024-04-24 10:00:47.738674] INFO [LIB] stat (utility.h:1197) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=67] [PALF STAT LOG IO WORKER QUEUE SIZE](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=1, this=0x14b1945fe1e8, extra_info="log_io_worker_num:1, cb_thread_pool_tg_id:297") [2024-04-24 10:00:47.738729] INFO [LIB] stat (utility.h:1197) [861568][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=110] [PALF STAT FLUSH CB COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=131, this=0x14b186b515e0, extra_info="palf_id:1") [2024-04-24 10:00:47.742819] INFO [PALF] inner_write_once_ (log_block_handler.cpp:411) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=317] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=12479294, this={dio_aligned_buf:{buf_write_offset:3746, buf_padding_size:350, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:3781707, truncate_used_ts:4735441}, log_block_size:67108864, dir_fd:203, io_fd:135}, aligned_buf_len=4096, aligned_buf="001000001868080800000000001000000000000000001000001878080800000000000000000000000000000000000808080400A0000000000000000000001000000000000FFFFFFFFFFFFFFFFFF01000000FFFFFFFFFFFFFFFFFF01000001FFFFFFFFFFFFFFFFFF01FFFFFFFFFFFFFFFFFF01FFFFFFFFFFFFFFFFFF0100000000000000000000FFFFFFFFFFFFFFFFFF01FFFFFFFFFFFFFFFFFF0100000000000100FFFFFFFFFFFFFFFFFF0100000000000000000000000001802000000944442D4D4F4E2D5252001844442D4D4F4E2D52522048482E4D492E535358464620414D001C44442D4D4F4E2D52522048482E4D492E535358464620414D20545A5200808080870101868080800004000000000001A78180800007018D808080001016032DFF062B30383A303000018A808080000F0A053FFF8080808701019180808000D04E16032DFF0944442D4D4F4E2D52520001A080808000D14E16032DFF1844442D4D4F4E2D52522048482E4D492E535358464620414D0001A480808000D24E16032DFF1C44442D4D4F4E2D52522048482E4D492E535358464620414D20545A52000186808080000905053FFF2D018A808080000E05053FFF808080800400EDFFFFFFFFFFFF •Ó Î", aligned_block_offset=12476416, buf_len=868, total_write_size_=194155, total_write_size_after_dio_=688128, ob_pwrite_used_ts=4856570, count=121, each_pwrite_cost=40136) [2024-04-24 10:00:47.742971] INFO [LIB] stat (utility.h:1197) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=114] [PALF STAT WRITE LOG COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=220486, this=0x14b0efb333b0, extra_info="palf_id:1") [2024-04-24 10:00:47.743004] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1712] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.743051] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=69][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:79579966}], scn_array=[{val:1713924046589452000, v:0}], curr_size=868, accum_size=12695, time_cost=1152831) [2024-04-24 10:00:47.743150] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=70] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=12695) [2024-04-24 10:00:47.743279] INFO [LIB] stat (utility.h:1214) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=98] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=197, this=0x14b176ffc530, extra_info="") [2024-04-24 10:00:47.743389] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=83][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="10.0.0.3:2882", server="10.0.0.3:2882", update_func={old_end_lsn:{lsn:79579966}, new_end_lsn:{lsn:79580834}, old_advance_time_us:1713924046579073, new_ack_time_us:1713924047743380, advance delay(us):1164307}) [2024-04-24 10:00:47.743478] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=78] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=6112) [2024-04-24 10:00:47.743816] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=116] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=110, this=0x14b0efb29470, extra_info="") [2024-04-24 10:00:47.744081] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=178] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=8, avg_log_batch_cnt=2, total_group_log_size=16662, avg_group_log_size=2082) [2024-04-24 10:00:47.744180] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=83][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67158551}], scn_array=[{val:1713924046605790000, v:0}], curr_size=178, accum_size=178, time_cost=628202) [2024-04-24 10:00:47.744234] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=60][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:165097495}], scn_array=[{val:1713924046579624000, v:0}], curr_size=1573, accum_size=1573, time_cost=10171) [2024-04-24 10:00:47.744258] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=87] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=27, this=0x14b0efb296a0, extra_info="palf_id:1") [2024-04-24 10:00:47.744391] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=99] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=363169, this=0x14b0efb298d0, extra_info="palf_id:1") [2024-04-24 10:00:47.744480] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=77] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=56694, avg_log_gen_to_submit_time=56879, avg_log_submit_to_flush_time=217888, avg_log_submit_to_slide_time=306289) [2024-04-24 10:00:47.744616] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=129][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=304849, log_task={header:{begin_lsn:{lsn:79579966}, end_lsn:{lsn:79580834}, log_id:304849, min_scn:{val:1713924046589452000, v:0}, max_scn:{val:1713924046589452000, v:0}, data_len:812, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:79579402}, data_checksum:3031200725, accum_checksum:2216280496, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924046589701, freeze_ts:1713924046589714, submit_ts:1713924046589857, flushed_ts:1713924047743373, gen_to_freeze cost time:13, gen_to_submit cost time:156, submit_to_flush cost time:1153516}, fs_cb_begin_ts=1713924047744212, log_life_time=1154511) [2024-04-24 10:00:47.744616] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=109] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=28, this=0x14b1470ab6a0, extra_info="palf_id:1") [2024-04-24 10:00:47.744685] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [874495][T1004_OB_SLOG][T1004][Y0-0000000000000000-0-0] [lt=106] Slow write(duration=697454, write_len=4096) [2024-04-24 10:00:47.744754] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=95] [PALF STAT FLUSH CB COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=384, this=0x14b0efb335e0, extra_info="palf_id:1") [2024-04-24 10:00:47.744733] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=95] [PALF STAT LOG LIFE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=651017, this=0x14b1470ab8d0, extra_info="palf_id:1") [2024-04-24 10:00:47.744771] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [874495][T1004_OB_SLOG][T1004][Y0-0000000000000000-0-0] [lt=77] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=93232, offset=21596172}, end_cursor:ObLogCursor{file_id=1, log_id=93233, offset=21596421}, is_inited:true, is_local:false, buf_size:8192, buf:0x14b19eefc050, len:2036, log_data_len:249, seq:93232, flush_finish:false, flush_ret:0}) [2024-04-24 10:00:47.744846] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=106] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=3, avg_log_gen_to_freeze_time=128304, avg_log_gen_to_submit_time=128446, avg_log_submit_to_flush_time=517663, avg_log_submit_to_slide_time=522571) [2024-04-24 10:00:47.744990] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=132][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=259312, log_task={header:{begin_lsn:{lsn:67158551}, end_lsn:{lsn:67158729}, log_id:259312, min_scn:{val:1713924046605790000, v:0}, max_scn:{val:1713924046605790001, v:0}, data_len:122, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:67158551}, data_checksum:1857683618, accum_checksum:2947771384, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924046791995, freeze_ts:1713924047115137, submit_ts:1713924047115323, flushed_ts:1713924047744541, gen_to_freeze cost time:323142, gen_to_submit cost time:323328, submit_to_flush cost time:629218}, fs_cb_begin_ts=1713924047744580, log_life_time=952585) [2024-04-24 10:00:47.745200] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874554][T1004_ApplySrv2][T1004][YB420A000003-000616C2BFC8F285-0-0] [lt=97][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924046589452000, v:0}, log_ts:{val:1713924046589452000, v:0}, lsn:{lsn:79580022}, submit_ts:1713924046589452}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176905}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1155730, ctx_lock_wait_time=9) [2024-04-24 10:00:47.745296] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [1559406][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262B-0-0] [lt=187][errcode=-4023] tx is 2pc logging(this={this:0x14b0fbedf050, ref:3, trans_id:{txid:338711}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924076767546, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046792910], ctx_create_time:1713924046778243}{ls_id:{id:1}, session_id:3221795357, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046789276, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046791807000, v:0}, log_ts:{val:1713924046791807000, v:0}, lsn:{lsn:298513878}, submit_ts:1713924046792973}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:338711}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338711}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046791807000, v:0}, start_scn:{val:1713924046791807000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924046791807000, v:0} commit_version={val:1713924046791807000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338711} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=215183838 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046791807000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046792973, block_frozen_memtable:null}) [2024-04-24 10:00:47.745404] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [874554][T1004_ApplySrv2][T1004][YB420A000003-000616C2BFC8F285-0-0] [lt=90] rpc statistics(total_trans_msg_count=4, total_batch_msg_count=0) [2024-04-24 10:00:47.745476] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [1559406][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262B-0-0] [lt=171][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fbedf050, ref:3, trans_id:{txid:338711}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924076767546, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046792910], ctx_create_time:1713924046778243}{ls_id:{id:1}, session_id:3221795357, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046789276, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046791807000, v:0}, log_ts:{val:1713924046791807000, v:0}, lsn:{lsn:298513878}, submit_ts:1713924046792973}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:338711}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338711}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046791807000, v:0}, start_scn:{val:1713924046791807000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924046791807000, v:0} commit_version={val:1713924046791807000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338711} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=215183838 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046791807000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046792973, block_frozen_memtable:null}) [2024-04-24 10:00:47.745524] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:188) [862224][T1002_FreInfoRe][T1002][Y0-0000000000000000-0-0] [lt=75] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1713922241263293000) [2024-04-24 10:00:47.745593] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [1559406][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262B-0-0] [lt=109][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338711}) [2024-04-24 10:00:47.745690] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [862224][T1002_FreInfoRe][T1002][Y0-0000000000000000-0-0] [lt=76] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, append size=9559) [2024-04-24 10:00:47.745702] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [1559406][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262B-0-0] [lt=97][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338711}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047739377, cluster_id:1713874979}, expire_ts:1713924076788074, commit_start_scn:{val:1713924046780604000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.745788] INFO [LIB] stat (utility.h:1197) [862224][T1002_FreInfoRe][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT APPEND COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x14b134390e50, extra_info="id:1001") [2024-04-24 10:00:47.745838] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [1559406][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262B-0-0] [lt=86][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338711}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047739377, cluster_id:1713874979}, expire_ts:1713924076788074, commit_start_scn:{val:1713924046780604000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.745853] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AB-0-0] [lt=101][errcode=-4023] tx is 2pc logging(this={this:0x14b147fffbd0, ref:4, trans_id:{txid:176906}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925046666492, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046708409], ctx_create_time:1713924046701144}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79581303}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046705015, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046707343000, v:0}, log_ts:{val:1713924046707343000, v:0}, lsn:{lsn:79582268}, submit_ts:1713924046708648}, this:0x14b148002b40, is_inited_:true, trans_id:{txid:176906}, ls_id:{id:1}, ctx:0x14b147fffbd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176906}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046707343000, v:0}, start_scn:{val:1713924046589452002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877459227536 max_table_version=1713877465782000 trans_version={val:1713924046707343000, v:0} commit_version={val:1713924046707343000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176906} ls_id=1 callback_alloc_count=5 callback_free_count=0 checksum=3165083175 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=8 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=8 unsynced_cnt=5 unsubmitted_cnt_=0 cb_statistics:[main=8, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046708648, block_frozen_memtable:null}) [2024-04-24 10:00:47.745091] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:426) [862397][T1001_FreInfoRe][T1001][Y0-0000000000000000-0-0] [lt=60] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2024-04-24 10:00:47.746085] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AB-0-0] [lt=225][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b147fffbd0, ref:4, trans_id:{txid:176906}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925046666492, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046708409], ctx_create_time:1713924046701144}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79581303}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046705015, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046707343000, v:0}, log_ts:{val:1713924046707343000, v:0}, lsn:{lsn:79582268}, submit_ts:1713924046708648}, this:0x14b148002b40, is_inited_:true, trans_id:{txid:176906}, ls_id:{id:1}, ctx:0x14b147fffbd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176906}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046707343000, v:0}, start_scn:{val:1713924046589452002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877459227536 max_table_version=1713877465782000 trans_version={val:1713924046707343000, v:0} commit_version={val:1713924046707343000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176906} ls_id=1 callback_alloc_count=5 callback_free_count=0 checksum=3165083175 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=8 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=8 unsynced_cnt=5 unsubmitted_cnt_=0 cb_statistics:[main=8, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046708648, block_frozen_memtable:null}) [2024-04-24 10:00:47.746326] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AB-0-0] [lt=233][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176906}) [2024-04-24 10:00:47.746433] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AB-0-0] [lt=99][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176906}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047739966, cluster_id:1713874979}, expire_ts:1713925046666035, commit_start_scn:{val:1713924046703599000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.746515] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AB-0-0] [lt=65][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176906}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047739966, cluster_id:1713874979}, expire_ts:1713925046666035, commit_start_scn:{val:1713924046703599000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.746757] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:159) [861548][OmtNodeBalancer][T1][YB420A000003-000616C2A8291461-0-0] [lt=78] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2024-04-24 10:00:47.746861] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [861548][OmtNodeBalancer][T1][YB420A000003-000616C2A8291461-0-0] [lt=65] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2024-04-24 10:00:47.746978] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:131) [861548][OmtNodeBalancer][T1][YB420A000003-000616C2A8291461-0-0] [lt=106] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2024-04-24 10:00:47.747059] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2326786][T1002_L0_G1][T1002][YB420A000003-000616C2B35ECB49-0-0] [lt=79] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="10.0.0.3:2882", committed size=3935) [2024-04-24 10:00:47.747210] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000003-000616C2B35ECB49-0-0] [lt=69] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=50, this=0x14b13adc36a0, extra_info="palf_id:1002") [2024-04-24 10:00:47.747295] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000003-000616C2B35ECB49-0-0] [lt=70] [PALF STAT LOG LIFE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=785108, this=0x14b13adc38d0, extra_info="palf_id:1002") [2024-04-24 10:00:47.747288] INFO [PALF] read (log_cache.cpp:100) [862285][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=111] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=3935, hit_cnt=2, read_cnt=2, hit rate=1.000000000000000000e+00) [2024-04-24 10:00:47.747399] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2326786][T1002_L0_G1][T1002][YB420A000003-000616C2B35ECB49-0-0] [lt=95] [PALF STAT LOG TASK TIME](palf_id=1002, self="10.0.0.3:2882", total_slide_log_cnt=2, avg_log_gen_to_freeze_time=55, avg_log_gen_to_submit_time=197, avg_log_submit_to_flush_time=775804, avg_log_submit_to_slide_time=784911) [2024-04-24 10:00:47.747425] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1208) [862285][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=111] succ to submit log task to replay service(to_submit_lsn={lsn:165096762}, to_submit_scn={val:1713924046271305000, v:0}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713924047747166, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:165096828}, next_to_submit_scn_:{val:1713924046271305001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1713875572331196000, v:0}, iterator_:{iterator_impl:{buf_:0x14b1144cd000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2362, log_storage_:{IteratorStorage:{start_lsn:{lsn:165096706}, end_lsn:{lsn:165099068}, read_buf:{buf_len_:24576, buf_:0x14b1144cd000}, block_size:67104768, log_storage_:0x14b13adc9470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:667, prev_entry_scn:{val:1713924046519555000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:635, scn_:{val:1713924046519555000, v:0}, data_checksum:2916439603, flag:1}}, init_mode_version:0, accumulate_checksum:1778579372, curr_entry_is_padding:0, padding_entry_size:733, padding_entry_scn:{val:1713924046519555000, v:0}}}}}) [2024-04-24 10:00:47.747475] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000003-000616C2B35ECB49-0-0] [lt=69][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=301030, log_task={header:{begin_lsn:{lsn:165096706}, end_lsn:{lsn:165097495}, log_id:301030, min_scn:{val:1713924046271305000, v:0}, max_scn:{val:1713924046519555000, v:0}, data_len:733, proposal_id:5, prev_lsn:{lsn:165095133}, prev_proposal_id:5, committed_end_lsn:{lsn:165096706}, data_checksum:1845477092, accum_checksum:1778579372, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924046575832, freeze_ts:1713924046575924, submit_ts:1713924046576004, flushed_ts:1713924047734328, gen_to_freeze cost time:92, gen_to_submit cost time:172, submit_to_flush cost time:1158324}, fs_cb_begin_ts=1713924047747154, log_life_time=1171322) [2024-04-24 10:00:47.747800] INFO [LIB] stat (utility.h:1197) [862286][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=83] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=265, this=0x14b16c6048f8, extra_info="") [2024-04-24 10:00:47.747934] INFO [LIB] stat (utility.h:1197) [862286][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=114] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=471, this=0x14b16c604b28, extra_info="") [2024-04-24 10:00:47.748203] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [862236][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=59] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924037336451, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924047746903, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:47.748211] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [862284][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=83] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924036997437, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924047746903, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:47.749680] INFO [LIB] stat (utility.h:1197) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=95] [PALF STAT WRITE LOG COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=401458, this=0x14b0c0f013b0, extra_info="palf_id:1002") [2024-04-24 10:00:47.749800] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=112][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:40356249}], scn_array=[{val:1713924046252158001, v:0}], curr_size=514, accum_size=870, time_cost=1157355) [2024-04-24 10:00:47.749835] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=54][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:314126934}], scn_array=[{val:1713924046407795000, v:0}], curr_size=1859, accum_size=1859, time_cost=13668) [2024-04-24 10:00:47.749948] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=136] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=870) [2024-04-24 10:00:47.749956] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [862230][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=145] Slow write(duration=615698, write_len=8192) [2024-04-24 10:00:47.750043] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [862230][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=78] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=101080, offset=22880143}, end_cursor:ObLogCursor{file_id=1, log_id=101081, offset=22880392}, is_inited:true, is_local:false, buf_size:8192, buf:0x14b19ef58050, len:4209, log_data_len:249, seq:101080, flush_finish:false, flush_ret:0}) [2024-04-24 10:00:47.750051] INFO [LIB] stat (utility.h:1214) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=68] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=1771, this=0x14b176ffd3b0, extra_info="") [2024-04-24 10:00:47.750121] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=68][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:1002185529}], scn_array=[{val:1713924046605790000, v:0}], curr_size=1573, accum_size=1573, time_cost=13419) [2024-04-24 10:00:47.750236] INFO [LIB] stat (utility.h:1197) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=333] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=281359, this=0x14b0c0e353b0, extra_info="palf_id:1001") [2024-04-24 10:00:47.750261] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [862263][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=76] cannot_recycle_log_size statistics(cannot_recycle_log_size=898919308, threshold=1739461755, need_update_checkpoint_scn=false) [2024-04-24 10:00:47.750325] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=78][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:88112990}], scn_array=[{val:1713924046589452001, v:0}], curr_size=122, accum_size=1115, time_cost=1085895) [2024-04-24 10:00:47.750402] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=71] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=1115) [2024-04-24 10:00:47.750536] INFO [LIB] stat (utility.h:1214) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=103] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=156, this=0x14b176ffcc70, extra_info="") [2024-04-24 10:00:47.750921] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:122) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=81] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2024-04-24 10:00:47.751460] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=65] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.751941] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2325892][T1_L0_G1][T1][YB420A000003-000616C2AA1E7FBF-0-0] [lt=121] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=5425) [2024-04-24 10:00:47.752192] INFO [LIB] stat (utility.h:1197) [2325892][T1_L0_G1][T1][YB420A000003-000616C2AA1E7FBF-0-0] [lt=81] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=89, this=0x14b186b476a0, extra_info="palf_id:1") [2024-04-24 10:00:47.752305] INFO [LIB] stat (utility.h:1197) [2325892][T1_L0_G1][T1][YB420A000003-000616C2AA1E7FBF-0-0] [lt=83] [PALF STAT LOG LIFE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=527559, this=0x14b186b478d0, extra_info="palf_id:1") [2024-04-24 10:00:47.752336] INFO [PALF] read (log_cache.cpp:100) [861714][T1_ReplaySrv6][T1][Y0-0000000000000000-0-0] [lt=74] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5425, hit_cnt=3, read_cnt=3, hit rate=1.000000000000000000e+00) [2024-04-24 10:00:47.752415] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2325892][T1_L0_G1][T1][YB420A000003-000616C2AA1E7FBF-0-0] [lt=103] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=3, avg_log_gen_to_freeze_time=50, avg_log_gen_to_submit_time=145, avg_log_submit_to_flush_time=517324, avg_log_submit_to_slide_time=527413) [2024-04-24 10:00:47.752522] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325892][T1_L0_G1][T1][YB420A000003-000616C2AA1E7FBF-0-0] [lt=101][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=364232, log_task={header:{begin_lsn:{lsn:179736232}, end_lsn:{lsn:179737277}, log_id:364232, min_scn:{val:1713924046351265000, v:0}, max_scn:{val:1713924046506830000, v:0}, data_len:989, proposal_id:5, prev_lsn:{lsn:179735217}, prev_proposal_id:5, committed_end_lsn:{lsn:179736232}, data_checksum:1070348233, accum_checksum:2809916098, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924046588609, freeze_ts:1713924046588659, submit_ts:1713924046588718, flushed_ts:1713924047738587, gen_to_freeze cost time:50, gen_to_submit cost time:109, submit_to_flush cost time:1149869}, fs_cb_begin_ts=1713924047752055, log_life_time=1163446) [2024-04-24 10:00:47.753141] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [861712][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=140] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924036222499, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924047753117, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:47.753257] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:159) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=70] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2024-04-24 10:00:47.753301] INFO [LIB] stat (utility.h:1197) [861712][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=94] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=186, this=0x14b1976048f8, extra_info="") [2024-04-24 10:00:47.753336] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [861660][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=124] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924036532523, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924047753117, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:47.753307] INFO [LIB] stat (utility.h:1197) [861715][T1_ReplaySrv7][T1][Y0-0000000000000000-0-0] [lt=113] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=995, this=0x14b197604b28, extra_info="") [2024-04-24 10:00:47.753945] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [861715][T1_ReplaySrv7][T1][Y0-0000000000000000-0-0] [lt=156] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924036799939, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924047753117, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:47.754267] INFO [SHARE] update_tenant_recovery_status_in_trans (ob_tenant_info_proxy.cpp:460) [874422][T1004_TeRec][T1004][YB420A000003-000616C2B5596E95-0-0] [lt=165] tenant scn gap info(IS_MAX_GAP_REACHED=false, REAL_GAP=205740000, MAX_GAP=900000000000, new_sync_scn={val:1713924044904071000, v:0}, new_replayable_scn={val:1713924044904071000, v:0}, new_readable_scn={val:1713924044698331000, v:0}, old_tenant_info={tenant_id:1004, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1713924043670180000, replayable_scn:1713924043670180000, readable_scn:1713924043670180000, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1002}) [2024-04-24 10:00:47.754893] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [874589][T1004_L2_G0][T1004][YB420A000002-000616C2C25C0151-0-0] [lt=86] get number(ret=0, service_type_=0, range=1, base_id=1713924047754365000, start_id=1713924047754365000, end_id=1713924047754365001) [2024-04-24 10:00:47.755131] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [874589][T1004_L2_G0][T1004][YB420A000002-000616C2C25C0151-0-0] [lt=88] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=17393) [2024-04-24 10:00:47.755317] INFO [LIB] stat (utility.h:1197) [874589][T1004_L2_G0][T1004][YB420A000002-000616C2C25C0151-0-0] [lt=164] [PALF STAT APPEND COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=0, this=0x14b0efafce50, extra_info="id:1") [2024-04-24 10:00:47.755528] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:196) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=119] refresh_resource_function_mapping_rule(tenant_id=1001, plan=, rules=[]) [2024-04-24 10:00:47.755582] WDIAG [STORAGE.TRANS] fill_redo_log (ob_memtable_context.cpp:655) [874589][T1004_L2_G0][T1004][YB420A000002-000616C2C25C0151-0-0] [lt=245][errcode=-5199] fill_redo_log fail(ret=-5199, trans_id="{txid:327180}", buf="muta°€"¼", buf_len=1964, buf_pos=0, pending_log_size=125, *this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465798592 trans_version={val:1713924047754365000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:327180} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=1 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=2 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}) [2024-04-24 10:00:47.755638] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:100) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=87] refresh resource mapping rule success(tenant_id=1001, plan=) [2024-04-24 10:00:47.755638] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [874422][T1004_TeRec][T1003][YB420A000003-000616C2B5596E95-0-0] [lt=134] get number(ret=0, service_type_=0, range=1, base_id=1713924047754365000, start_id=1713924047754365000, end_id=1713924047754365001) [2024-04-24 10:00:47.755738] WDIAG [STORAGE.TRANS] fill_redo_log_ (ob_trans_part_ctx.cpp:2673) [874589][T1004_L2_G0][T1004][YB420A000002-000616C2C25C0151-0-0] [lt=143][errcode=-5199] fill redo log failed(ret=-5199, ret="OB_ERR_TOO_BIG_ROWSIZE", *this={this:0x14b148003290, ref:3, trans_id:{txid:327180}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924077752956, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924047754828], ctx_create_time:1713924047753117}{ls_id:{id:1}, session_id:3221542000, part_trans_action:2, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79585588}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.3:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924047751949, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:0}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b148006200, is_inited_:true, trans_id:{txid:327180}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:327180}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924047754365000, v:0}, start_scn:{val:1713924046872416001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465798592 trans_version={val:1713924047754365000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:327180} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=1 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=2 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046872416001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924047754365, block_frozen_memtable:null}) [2024-04-24 10:00:47.756000] WDIAG [STORAGE.TRANS] submit_redo_log_ (ob_trans_part_ctx.cpp:2765) [874589][T1004_L2_G0][T1004][YB420A000002-000616C2C25C0151-0-0] [lt=255][errcode=-5199] encounter too big row size error, need extend log buffer(ret=-5199, *this={this:0x14b148003290, ref:3, trans_id:{txid:327180}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924077752956, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924047754828], ctx_create_time:1713924047753117}{ls_id:{id:1}, session_id:3221542000, part_trans_action:2, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79585588}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.3:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924047751949, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:0}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b148006200, is_inited_:true, trans_id:{txid:327180}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:327180}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924047754365000, v:0}, start_scn:{val:1713924046872416001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465798592 trans_version={val:1713924047754365000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:327180} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=1 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=2 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046872416001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924047754365, block_frozen_memtable:null}) [2024-04-24 10:00:47.756207] INFO [STORAGE.TRANS] submit_redo_log_ (ob_trans_part_ctx.cpp:2779) [874589][T1004_L2_G0][T1004][YB420A000002-000616C2C25C0151-0-0] [lt=155] extend log buffer success(*this={this:0x14b148003290, ref:3, trans_id:{txid:327180}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924077752956, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924047754828], ctx_create_time:1713924047753117}{ls_id:{id:1}, session_id:3221542000, part_trans_action:2, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79585588}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.3:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924047751949, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:0}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b148006200, is_inited_:true, trans_id:{txid:327180}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:327180}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924047754365000, v:0}, start_scn:{val:1713924046872416001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465798592 trans_version={val:1713924047754365000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:327180} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=1 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=2 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046872416001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924047754365, block_frozen_memtable:null}, log_block={fill_buf_:{buf:0x14b045204030, default_buf:0x14b0fd3ce798, len:1965056}, replay_buf_:null, len_:1965056, pos_:54, cur_log_type_:0, cb_arg_array_:[], big_segment_buf_:NULL}) [2024-04-24 10:00:47.758513] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B47F14AC-0-0] [lt=94] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=10, this=0x14b13adc3470, extra_info="") [2024-04-24 10:00:47.758653] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B47F14AC-0-0] [lt=80] [PALF STAT GROUP LOG INFO](palf_id=1002, self="10.0.0.3:2882", role="FOLLOWER", total_group_log_cnt=3, avg_log_batch_cnt=0, total_group_log_size=2484, avg_group_log_size=828) [2024-04-24 10:00:47.758755] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=75] [PALF STAT FLUSH CB COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=92, this=0x14b0c0f015e0, extra_info="palf_id:1002") [2024-04-24 10:00:47.759012] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=179][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1001, self="10.0.0.3:2882", server="10.0.0.3:2882", update_func={old_end_lsn:{lsn:88112990}, new_end_lsn:{lsn:88113112}, old_advance_time_us:1713924046570364, new_ack_time_us:1713924047758981, advance delay(us):1188617}) [2024-04-24 10:00:47.759048] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.759106] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=79] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="10.0.0.3:2882", committed size=1350) [2024-04-24 10:00:47.759268] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=68] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=65, this=0x14b0c0e2b470, extra_info="") [2024-04-24 10:00:47.759392] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=79] [PALF STAT GROUP LOG INFO](palf_id=1001, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=6, avg_log_batch_cnt=1, total_group_log_size=2806, avg_group_log_size=467) [2024-04-24 10:00:47.759551] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=103] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=20, this=0x14b0c0e2b6a0, extra_info="palf_id:1001") [2024-04-24 10:00:47.759762] INFO [LIB] stat (utility.h:1197) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AAFC06A0-0-0] [lt=92] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=13, this=0x14b186b47470, extra_info="") [2024-04-24 10:00:47.759650] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=70] [PALF STAT LOG LIFE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=370024, this=0x14b0c0e2b8d0, extra_info="palf_id:1001") [2024-04-24 10:00:47.759946] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [862260][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=90] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1713924045845177000, v:0}, total_part_count:3, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1914746, in_cluster_service=true, cluster_version={val:1713924045845177000, v:0}, min_cluster_version={val:1713924045845177000, v:0}, max_cluster_version={val:1713924045845177000, v:0}, get_cluster_version_err=0, cluster_version_delta=1914746, cluster_service_master="10.0.0.3:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=335063, succ_cluster_heartbeat_count=12, cluster_heartbeat_interval=50000, local_cluster_version={val:1713923632940224000, v:0}, local_cluster_delta=414819699, force_self_check=false, weak_read_refresh_interval=100000) [2024-04-24 10:00:47.759958] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=296] [PALF STAT LOG TASK TIME](palf_id=1001, self="10.0.0.3:2882", total_slide_log_cnt=6, avg_log_gen_to_freeze_time=67806, avg_log_gen_to_submit_time=67952, avg_log_submit_to_flush_time=283647, avg_log_submit_to_slide_time=302072) [2024-04-24 10:00:47.759984] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AAFC06A0-0-0] [lt=184] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="FOLLOWER", total_group_log_cnt=5, avg_log_batch_cnt=0, total_group_log_size=4912, avg_group_log_size=982) [2024-04-24 10:00:47.760056] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=89][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=330718, log_task={header:{begin_lsn:{lsn:88112990}, end_lsn:{lsn:88113112}, log_id:330718, min_scn:{val:1713924046589452001, v:0}, max_scn:{val:1713924046589452001, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:88112990}, data_checksum:2222604892, accum_checksum:1535067047, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924046664036, freeze_ts:1713924046664059, submit_ts:1713924046664189, flushed_ts:1713924047758975, gen_to_freeze cost time:23, gen_to_submit cost time:153, submit_to_flush cost time:1094786}, fs_cb_begin_ts=1713924047759512, log_life_time=1095476) [2024-04-24 10:00:47.760218] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=114] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=349, this=0x14b0c0e355e0, extra_info="palf_id:1001") [2024-04-24 10:00:47.760715] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2325522][T1004_L0_G1][T1004][YB420A000003-000616C2BA9E3145-0-0] [lt=71] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="10.0.0.3:2882", committed size=870) [2024-04-24 10:00:47.760942] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000003-000616C2BA9E3145-0-0] [lt=95] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=45, this=0x14b0c0ef76a0, extra_info="palf_id:1002") [2024-04-24 10:00:47.761050] INFO [PALF] read (log_cache.cpp:100) [874558][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=146] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=870, hit_cnt=4, read_cnt=4, hit rate=1.000000000000000000e+00) [2024-04-24 10:00:47.761058] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000003-000616C2BA9E3145-0-0] [lt=100] [PALF STAT LOG LIFE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=432623, this=0x14b0c0ef78d0, extra_info="palf_id:1002") [2024-04-24 10:00:47.760769] INFO [LIB] stat (utility.h:1197) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2C208F1CE-0-0] [lt=243] [APPLY STAT CB APPEND COST TIME](cur_stat_count=13, stat_interval=5000000, avg (count/cost)=204, this=0x14b0c0e3c280, extra_info="") [2024-04-24 10:00:47.761202] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [861648][T1_ReqMemEvict][T1][YB420A000003-000616C2A96914B3-0-0] [lt=133] slow inner sql(last_ret=0, sql={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, process_time=1252727) [2024-04-24 10:00:47.761254] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2325522][T1004_L0_G1][T1004][YB420A000003-000616C2BA9E3145-0-0] [lt=175] [PALF STAT LOG TASK TIME](palf_id=1002, self="10.0.0.3:2882", total_slide_log_cnt=5, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=79, avg_log_submit_to_flush_time=368673, avg_log_submit_to_slide_time=432543) [2024-04-24 10:00:47.761300] INFO [LIB] stat (utility.h:1197) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2C208F1CE-0-0] [lt=519] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=13, stat_interval=5000000, avg (count/cost)=188066, this=0x14b0c0e3c4b0, extra_info="") [2024-04-24 10:00:47.761336] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000003-000616C2BA9E3145-0-0] [lt=73][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=299911, log_task={header:{begin_lsn:{lsn:40356249}, end_lsn:{lsn:40356763}, log_id:299911, min_scn:{val:1713924046252158001, v:0}, max_scn:{val:1713924046271305001, v:0}, data_len:458, proposal_id:5, prev_lsn:{lsn:40356137}, prev_proposal_id:5, committed_end_lsn:{lsn:40356249}, data_checksum:2768610552, accum_checksum:3120698878, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924046592096, freeze_ts:1713924046592105, submit_ts:1713924046592165, flushed_ts:1713924047758653, gen_to_freeze cost time:9, gen_to_submit cost time:69, submit_to_flush cost time:1166488}, fs_cb_begin_ts=1713924047760821, log_life_time=1168725) [2024-04-24 10:00:47.761416] INFO [LIB] stat (utility.h:1197) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2C208F1CE-0-0] [lt=94] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=13, stat_interval=5000000, avg (count/cost)=0, this=0x14b0c0e3c6e0, extra_info="") [2024-04-24 10:00:47.761532] INFO [LIB] stat (utility.h:1197) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2C208F1CE-0-0] [lt=103] [APPLY STAT CB EXECUTE TIME](cur_stat_count=13, stat_interval=5000000, avg (count/cost)=-617, this=0x14b0c0e3c910, extra_info="") [2024-04-24 10:00:47.761607] INFO [LIB] stat (utility.h:1197) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2C208F1CE-0-0] [lt=63] [APPLY STAT CB TOTAL TIME](cur_stat_count=13, stat_interval=5000000, avg (count/cost)=187653, this=0x14b0c0e3cb40, extra_info="") [2024-04-24 10:00:47.761713] WDIAG [CLOG] statistics_cb_cost_ (ob_log_apply_service.cpp:933) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2C208F1CE-0-0] [lt=96][errcode=-4389] cb cost too much time(lsn={lsn:88113046}, scn={val:1713924046589452001, v:0}, idx=1, total_cost_time=1093537, append_cost_time=200, cb_wait_thread_time=1096535, cb_wait_commit_time=0, cb_cost_time=-3198, append_start_time=1713924046664018, append_finish_time=1713924046664218, cb_first_handle_time=1713924047760753, cb_finish_time=1713924047757555) [2024-04-24 10:00:47.762063] INFO [LIB] stat (utility.h:1197) [874501][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=85] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=36, stat_interval=1000000, avg (count/cost)=798, this=0x14b1312048f8, extra_info="") [2024-04-24 10:00:47.762186] INFO [LIB] stat (utility.h:1197) [874501][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=100] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=36, stat_interval=1000000, avg (count/cost)=40, this=0x14b131204b28, extra_info="") [2024-04-24 10:00:47.765751] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:426) [861646][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=79] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2024-04-24 10:00:47.766318] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F295-0-0] [lt=106] get number(ret=0, service_type_=0, range=1, base_id=1713924047765133000, start_id=1713924047765133000, end_id=1713924047765133001) [2024-04-24 10:00:47.768186] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:860) [882037][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=88] memtable handle rollback to successfuly(from_seq_no=1713924047767922, to_seq_no=1713924047758610, *this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877457} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=1, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}) [2024-04-24 10:00:47.768445] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [882037][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=142] rpc statistics(total_trans_msg_count=12, total_batch_msg_count=0) [2024-04-24 10:00:47.771076] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=75] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.771183] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=89] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.771272] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=83] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.773516] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=112][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:165099068}], scn_array=[{val:1713924047274777000, v:0}], curr_size=122, accum_size=1695, time_cost=14767) [2024-04-24 10:00:47.778123] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [875191][T1004_L0_G5][T1004][YB420A000002-000616C2BEDDD113-0-0] [lt=102] get number(ret=0, service_type_=0, range=1, base_id=1713924047777372000, start_id=1713924047777372000, end_id=1713924047777372001) [2024-04-24 10:00:47.782681] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [874745][T1003_ArcSender][T1003][YB420A000003-000616C2B638F0E7-0-0] [lt=119][errcode=-4018] try free send task failed(ret=-4018) [2024-04-24 10:00:47.783525] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2C20C019D-0-0] [lt=89] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=9, this=0x14b0c0ef7470, extra_info="") [2024-04-24 10:00:47.783711] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2C20C019D-0-0] [lt=143] [PALF STAT GROUP LOG INFO](palf_id=1002, self="10.0.0.3:2882", role="FOLLOWER", total_group_log_cnt=4, avg_log_batch_cnt=0, total_group_log_size=1197, avg_group_log_size=299) [2024-04-24 10:00:47.784331] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=123] new mysql sessid created(conn.sessid_=3221542053, support_ssl=false) [2024-04-24 10:00:47.784491] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=108] sm conn init succ(conn.sessid_=3221542053, sess.client_addr_="10.0.0.3:48118") [2024-04-24 10:00:47.784596] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=91] accept one succ(*s={this:0x14b1941f4030, session_id:3221542053, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:168}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:47.785045] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:943) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=153][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x14b1941f4030, session_id:3221542053, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:168}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:47.785236] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=182] kill and revert session(conn.sessid_=3221542053, proxy_sessid=0, server_id=1, ret=0) [2024-04-24 10:00:47.785380] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:918) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=130] sock ref clean, do destroy(*s={this:0x14b1941f4030, session_id:3221542053, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:168}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:47.785495] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:223) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=104] mark session id unused(conn.sessid_=3221542053) [2024-04-24 10:00:47.785604] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=99] connection close(sessid=3221542053, proxy_sessid=0, tenant_id=0, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:0], disconnect_state=0) [2024-04-24 10:00:47.791860] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=61][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:1002187214}], scn_array=[{val:1713924047777372000, v:0}], curr_size=1572, accum_size=3257, time_cost=12995) [2024-04-24 10:00:47.800999] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=57] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.801104] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=83] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.801204] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=94] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.807259] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [874394][T1004_Occam][T1004][YB420A000003-000616C2B6C982E6-0-0] [lt=96] for each ls(succ_num=3, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:47.812659] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=100][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67158729}], scn_array=[{val:1713924047777372000, v:0}], curr_size=122, accum_size=300, time_cost=11549) [2024-04-24 10:00:47.825935] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=83][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.833983] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=66] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-04-24 10:00:47.836989] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [861692][T1_LSArchiveMgr][T1][YB420A000003-000616C2A888F0E7-0-0] [lt=86] gc stale ls task succ [2024-04-24 10:00:47.836998] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:788) [862091][T1_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=96] ps cache evict(tenant_id=1, stmt_id_map_.size()=15, stmt_info_map_.size()=15, op.get_used_size()=32344, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=15) [2024-04-24 10:00:47.837939] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=130] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1001, min_version={val:1713924045694942002, v:0}, base_version={val:1713924045694942002, v:0}, server_count=2, svr_array=[{addr:"10.0.0.3:2882", version:{val:1713924045694942002, v:0}, valid_part_count:0, total_part_count:1, generate_tstamp:1713924047837785, is_skipped:true}, {addr:"10.0.0.2:2882", version:{val:1713924046050326000, v:0}, valid_part_count:0, total_part_count:1, generate_tstamp:1713924047810852, is_skipped:true}], skip_server_count=2, skip_servers=["10.0.0.3:2882", "10.0.0.2:2882"]) [2024-04-24 10:00:47.838058] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=109] no need to update min/max version(new_version={val:1713924045694942002, v:0}, min_version={val:1713924045694942002, v:0}, max_version={val:1713924045694942002, v:0}) [2024-04-24 10:00:47.842247] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [874542][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=183] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1001}, {id:1}]) [2024-04-24 10:00:47.843179] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=77] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.844100] INFO [CLOG] is_in_sync (ob_log_handler.cpp:546) [861725][T1_L5_G0][T1][YB420A000002-000616C2A8DC429C-0-0] [lt=124] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2024-04-24 10:00:47.844594] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_ls_info.cpp:54) [861725][T1_L5_G0][T1][YB420A000002-000616C2A8DC429C-0-0] [lt=77][errcode=-4008] execute fail(ret=-4008) [2024-04-24 10:00:47.844776] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AC-0-0] [lt=83][errcode=-4023] tx is 2pc logging(this={this:0x14b055e1a050, ref:3, trans_id:{txid:176904}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138442, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589701], ctx_create_time:1713924046585672}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046585674, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452001, v:0}, log_ts:{val:1713924046589452001, v:0}, lsn:{lsn:79580890}, submit_ts:1713924046589452}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176904}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176904}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452001, v:0}, start_scn:{val:1713924046589452001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877457407528 trans_version={val:1713924046589452001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176904} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3140423830 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:47.845012] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AC-0-0] [lt=226][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e1a050, ref:3, trans_id:{txid:176904}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138442, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589701], ctx_create_time:1713924046585672}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046585674, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452001, v:0}, log_ts:{val:1713924046589452001, v:0}, lsn:{lsn:79580890}, submit_ts:1713924046589452}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176904}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176904}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452001, v:0}, start_scn:{val:1713924046589452001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877457407528 trans_version={val:1713924046589452001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176904} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3140423830 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:47.845202] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AC-0-0] [lt=177][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176904}) [2024-04-24 10:00:47.845277] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AC-0-0] [lt=69][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176904}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924047840006, cluster_id:1713874979}, expire_ts:1713925045138403, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.845371] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AC-0-0] [lt=78][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176904}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924047840006, cluster_id:1713874979}, expire_ts:1713925045138403, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.851711] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=110] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.855293] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E96-0-0] [lt=92][errcode=-4023] tx is 2pc logging(this={this:0x14b168f2d710, ref:3, trans_id:{txid:877456}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924646597291, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046606334], ctx_create_time:1713924046597291}{ls_id:{id:1001}, session_id:3221852891, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046603230, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046605790000, v:0}, log_ts:{val:1713924046605790000, v:0}, lsn:{lsn:1002185585}, submit_ts:1713924046605790}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877456}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877456}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046605790000, v:0}, start_scn:{val:1713924046605790000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924046605790000, v:0} commit_version={val:1713924046605790000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877456} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=737308951 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046605790000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046605790, block_frozen_memtable:null}) [2024-04-24 10:00:47.855515] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [874436][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=95] ====== tenant freeze timer task ====== [2024-04-24 10:00:47.855490] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E96-0-0] [lt=182][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b168f2d710, ref:3, trans_id:{txid:877456}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924646597291, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046606334], ctx_create_time:1713924046597291}{ls_id:{id:1001}, session_id:3221852891, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046603230, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046605790000, v:0}, log_ts:{val:1713924046605790000, v:0}, lsn:{lsn:1002185585}, submit_ts:1713924046605790}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877456}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877456}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046605790000, v:0}, start_scn:{val:1713924046605790000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924046605790000, v:0} commit_version={val:1713924046605790000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877456} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=737308951 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046605790000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046605790, block_frozen_memtable:null}) [2024-04-24 10:00:47.855651] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E96-0-0] [lt=151][errcode=-4023] commit fail(ret=-4023, coord={id:1001}, tx_id={txid:877456}) [2024-04-24 10:00:47.855778] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E96-0-0] [lt=119][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877456}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924047850506, cluster_id:1713874979}, expire_ts:1713924558320000, commit_start_scn:{val:1713924046594982000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:47.855859] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E96-0-0] [lt=65][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877456}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924047850506, cluster_id:1713874979}, expire_ts:1713924558320000, commit_start_scn:{val:1713924046594982000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:47.857677] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [862417][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=62] report RowHolderMapper summary info(count=5, bkt_cnt=248) [2024-04-24 10:00:47.859223] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=87][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.859361] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1073) [862237][T1002_ReplayPro][T1002][Y0-0000000000000000-0-0] [lt=128] replay status is not follower(min_unreplayed_lsn={lsn:64863204}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713921656489962, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:64863204}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:64863204}, end_lsn:{lsn:64863204}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x14b1470b1470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-04-24 10:00:47.859520] INFO [CLOG] operator() (ob_log_replay_service.cpp:1590) [862237][T1002_ReplayPro][T1002][Y0-0000000000000000-0-0] [lt=132] get_replay_process success(id={id:1}, replayed_log_size=64863204, unreplayed_log_size=0) [2024-04-24 10:00:47.859603] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1073) [862237][T1002_ReplayPro][T1002][Y0-0000000000000000-0-0] [lt=71] replay status is not follower(min_unreplayed_lsn={lsn:951799104}, base_lsn={lsn:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713921656899305, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:4063, ret_code_:0}}, next_to_submit_lsn_:{lsn:951799104}, next_to_submit_scn_:{val:1713900501780579001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1713875572267468000, v:0}, iterator_:{iterator_impl:{buf_:0x14b0ec177000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:951799104}, end_lsn:{lsn:951799104}, read_buf:{buf_len_:8192, buf_:0x14b0ec177000}, block_size:67104768, log_storage_:0x14b1336f1470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-04-24 10:00:47.859797] INFO [CLOG] operator() (ob_log_replay_service.cpp:1590) [862237][T1002_ReplayPro][T1002][Y0-0000000000000000-0-0] [lt=183] get_replay_process success(id={id:1001}, replayed_log_size=951799104, unreplayed_log_size=0) [2024-04-24 10:00:47.859922] INFO [CLOG] operator() (ob_log_replay_service.cpp:1590) [862237][T1002_ReplayPro][T1002][Y0-0000000000000000-0-0] [lt=117] get_replay_process success(id={id:1002}, replayed_log_size=165099068, unreplayed_log_size=0) [2024-04-24 10:00:47.860010] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:159) [862237][T1002_ReplayPro][T1002][Y0-0000000000000000-0-0] [lt=58] dump tenant replay process(tenant_id=1002, unreplayed_log_size(MB)=0, estimate_time(second)=0, replayed_log_size(MB)=1127, last_replayed_log_size(MB)=1126, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2024-04-24 10:00:47.860073] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [862260][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=101] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1002, min_version={val:1713924045845177000, v:0}, base_version={val:1713924045845177000, v:0}, server_count=2, svr_array=[{addr:"10.0.0.3:2882", version:{val:1713924045845177000, v:0}, valid_part_count:0, total_part_count:3, generate_tstamp:1713924047760056, is_skipped:true}, {addr:"10.0.0.2:2882", version:{val:1713924046512037000, v:0}, valid_part_count:0, total_part_count:3, generate_tstamp:1713924047774662, is_skipped:true}], skip_server_count=2, skip_servers=["10.0.0.3:2882", "10.0.0.2:2882"]) [2024-04-24 10:00:47.860174] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [862260][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=76] no need to update min/max version(new_version={val:1713924045845177000, v:0}, min_version={val:1713924045845177000, v:0}, max_version={val:1713924045845177000, v:0}) [2024-04-24 10:00:47.868933] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [861689][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=119] ====== check clog disk timer task ====== [2024-04-24 10:00:47.869055] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [861689][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=108] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=235) [2024-04-24 10:00:47.869157] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [861689][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=69] cannot_recycle_log_size statistics(cannot_recycle_log_size=112634333, threshold=644245094, need_update_checkpoint_scn=false) [2024-04-24 10:00:47.871295] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=111] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.871667] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=91] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.871777] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=101] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.893987] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2609-0-0] [lt=72][errcode=-4023] tx is 2pc logging(this={this:0x14b15a16b090, ref:3, trans_id:{txid:391371}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924076773625, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046800195], ctx_create_time:1713924046784446}{ls_id:{id:1}, session_id:3221795340, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046791808, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046799990000, v:0}, log_ts:{val:1713924046799990000, v:0}, lsn:{lsn:314127056}, submit_ts:1713924046799990}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:391371}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:391371}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046799990000, v:0}, start_scn:{val:1713924046799990000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875442714112 max_table_version=1713875442714112 trans_version={val:1713924046799990000, v:0} commit_version={val:1713924046799990000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:391371} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=2612980561 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046799990000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046799990, block_frozen_memtable:null}) [2024-04-24 10:00:47.894156] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2609-0-0] [lt=158][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b15a16b090, ref:3, trans_id:{txid:391371}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924076773625, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046800195], ctx_create_time:1713924046784446}{ls_id:{id:1}, session_id:3221795340, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046791808, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046799990000, v:0}, log_ts:{val:1713924046799990000, v:0}, lsn:{lsn:314127056}, submit_ts:1713924046799990}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:391371}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:391371}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046799990000, v:0}, start_scn:{val:1713924046799990000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875442714112 max_table_version=1713875442714112 trans_version={val:1713924046799990000, v:0} commit_version={val:1713924046799990000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:391371} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=2612980561 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046799990000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046799990, block_frozen_memtable:null}) [2024-04-24 10:00:47.894268] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2609-0-0] [lt=105][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:391371}) [2024-04-24 10:00:47.894367] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2609-0-0] [lt=89][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1001, tx_id:{txid:391371}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047889432, cluster_id:1713874979}, expire_ts:1713924076795547, commit_start_scn:{val:1713924046786976000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.894507] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C2609-0-0] [lt=106][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1001, tx_id:{txid:391371}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047889432, cluster_id:1713874979}, expire_ts:1713924076795547, commit_start_scn:{val:1713924046786976000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.895681] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [861461][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=140] skip inspect bad block(last_check_time=1713875167993818, last_macro_idx=-1) [2024-04-24 10:00:47.900964] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=129] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.901322] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=84] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.901408] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=78] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.907525] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=96] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8890687488, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:47.912567] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=121] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1713924047411253], gts:1713924047408026000, latest_srr:[mts=1713924047912436]}) [2024-04-24 10:00:47.912720] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=137] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-24 10:00:47.912837] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [861494][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=81] get number(ret=0, service_type_=0, range=1, base_id=1713924047911690000, start_id=1713924047911690000, end_id=1713924047911690001) [2024-04-24 10:00:47.912972] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [861494][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=112] ts response task statistics(alloc_count=2169722, free_count=1084864) [2024-04-24 10:00:47.921217] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [862253][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=126] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=1, total_destructed_detector=1, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x14b168fe42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-04-24 10:00:47.924456] INFO pn_ratelimit (group.c:551) [861521][IngressService][T0][Y0-0000000000000000-0-0] [lt=155] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-04-24 10:00:47.926097] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=77][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.926969] INFO eloop_run (eloop.c:143) [861404][pnio2][T0][Y0-0000000000000000-0-0] [lt=120] PNIO [ratelimit] time: 1713924047926961, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007519, add_bytes: 0 [2024-04-24 10:00:47.940328] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_ls_info.cpp:54) [861725][T1_L5_G0][T1][YB420A000003-000616C2A8E93204-0-0] [lt=125][errcode=-4008] execute fail(ret=-4008) [2024-04-24 10:00:47.942555] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [862266][T1002_LSArchive][T1002][YB420A000003-000616C2AE58F0E7-0-0] [lt=85] gc stale ls task succ [2024-04-24 10:00:47.943300] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=74] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.944731] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AD-0-0] [lt=120][errcode=-4023] tx is 2pc logging(this={this:0x14b147fffbd0, ref:4, trans_id:{txid:176906}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925046666492, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046708409], ctx_create_time:1713924046701144}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79581303}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046705015, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046707343000, v:0}, log_ts:{val:1713924046707343000, v:0}, lsn:{lsn:79582268}, submit_ts:1713924046708648}, this:0x14b148002b40, is_inited_:true, trans_id:{txid:176906}, ls_id:{id:1}, ctx:0x14b147fffbd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176906}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046707343000, v:0}, start_scn:{val:1713924046589452002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877459227536 max_table_version=1713877465782000 trans_version={val:1713924046707343000, v:0} commit_version={val:1713924046707343000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176906} ls_id=1 callback_alloc_count=5 callback_free_count=0 checksum=3165083175 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=8 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=8 unsynced_cnt=5 unsubmitted_cnt_=0 cb_statistics:[main=8, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046708648, block_frozen_memtable:null}) [2024-04-24 10:00:47.944969] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AD-0-0] [lt=226][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b147fffbd0, ref:4, trans_id:{txid:176906}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925046666492, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046708409], ctx_create_time:1713924046701144}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79581303}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046705015, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046707343000, v:0}, log_ts:{val:1713924046707343000, v:0}, lsn:{lsn:79582268}, submit_ts:1713924046708648}, this:0x14b148002b40, is_inited_:true, trans_id:{txid:176906}, ls_id:{id:1}, ctx:0x14b147fffbd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176906}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046707343000, v:0}, start_scn:{val:1713924046589452002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877459227536 max_table_version=1713877465782000 trans_version={val:1713924046707343000, v:0} commit_version={val:1713924046707343000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176906} ls_id=1 callback_alloc_count=5 callback_free_count=0 checksum=3165083175 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=8 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=8 unsynced_cnt=5 unsubmitted_cnt_=0 cb_statistics:[main=8, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046708648, block_frozen_memtable:null}) [2024-04-24 10:00:47.945137] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AD-0-0] [lt=159][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176906}) [2024-04-24 10:00:47.945214] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AD-0-0] [lt=71][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176906}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047940140, cluster_id:1713874979}, expire_ts:1713925046666035, commit_start_scn:{val:1713924046703599000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.945324] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AD-0-0] [lt=88][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176906}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924047940140, cluster_id:1713874979}, expire_ts:1713925046666035, commit_start_scn:{val:1713924046703599000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:47.947323] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [861742][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=137] ls blacklist refresh finish(cost_time=31013) [2024-04-24 10:00:47.952114] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=149] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.959347] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=75][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:47.971425] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=71] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:47.971565] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=107] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:47.971717] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=125] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:47.975690] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B4AC6E61-0-0] [lt=139][errcode=-4023] tx is 2pc logging(this={this:0x14b168f30dd0, ref:3, trans_id:{txid:877457}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924647763173, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924047778260], ctx_create_time:1713924047763173}{ls_id:{id:1001}, session_id:3221998686, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924047770485, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924047777372000, v:0}, log_ts:{val:1713924047777372000, v:0}, lsn:{lsn:1002187270}, submit_ts:1713924047777372}, this:0x14b168f33d40, is_inited_:true, trans_id:{txid:877457}, ls_id:{id:1001}, ctx:0x14b168f30dd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877457}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924047777372000, v:0}, start_scn:{val:1713924047777372000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924047777372000, v:0} commit_version={val:1713924047777372000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877457} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=106315759 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924047777372000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924047778783, block_frozen_memtable:null}) [2024-04-24 10:00:47.975972] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B4AC6E61-0-0] [lt=275][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b168f30dd0, ref:3, trans_id:{txid:877457}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924647763173, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924047778260], ctx_create_time:1713924047763173}{ls_id:{id:1001}, session_id:3221998686, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924047770485, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924047777372000, v:0}, log_ts:{val:1713924047777372000, v:0}, lsn:{lsn:1002187270}, submit_ts:1713924047777372}, this:0x14b168f33d40, is_inited_:true, trans_id:{txid:877457}, ls_id:{id:1001}, ctx:0x14b168f30dd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877457}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924047777372000, v:0}, start_scn:{val:1713924047777372000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924047777372000, v:0} commit_version={val:1713924047777372000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877457} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=106315759 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924047777372000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924047778783, block_frozen_memtable:null}) [2024-04-24 10:00:47.976147] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B4AC6E61-0-0] [lt=167][errcode=-4023] commit fail(ret=-4023, coord={id:1001}, tx_id={txid:877457}) [2024-04-24 10:00:47.976245] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B4AC6E61-0-0] [lt=91][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877457}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924047971081, cluster_id:1713874979}, expire_ts:1713924618349599, commit_start_scn:{val:1713924047757555000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:47.976330] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B4AC6E61-0-0] [lt=68][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877457}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924047971081, cluster_id:1713874979}, expire_ts:1713924618349599, commit_start_scn:{val:1713924047757555000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:47.994389] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2720) [861318][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=71][errcode=-4002] get invalid Ethernet speed, use default(devname="eth0") [2024-04-24 10:00:47.994538] WDIAG [SERVER] runTimerTask (ob_server.cpp:3239) [861318][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=140][errcode=-4002] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4002, ret="OB_INVALID_ARGUMENT") [2024-04-24 10:00:47.999357] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [862447][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=65] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:48.001056] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=70] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.001168] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=96] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.001241] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=63] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.001407] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=33] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200876}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.002071] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200048}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.002577] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201384}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.003206] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200608}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.003731] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200450}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.004327] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=181] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200810}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.004923] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201394}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.005377] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201250}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.005960] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201318}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.006501] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200822}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.007104] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200874}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.007663] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200880}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.008240] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201270}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.008743] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:1152921504606846994}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.009391] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201434}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.009944] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201126}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.010513] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200124}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.010998] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200976}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.011519] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200566}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.012135] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201450}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.012729] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200312}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.013264] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200690}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.013836] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200808}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.014498] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=161] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200130}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.015116] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=115] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201138}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.015741] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200314}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.016375] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200031}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.017090] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=142] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200662}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.017641] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200492}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.018129] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=98] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200678}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.018715] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200332}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.019326] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201114}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.019975] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200304}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.020516] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200300}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.021082] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:1152921504606846997}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.021576] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200206}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.022121] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200410}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.022520] INFO [PALF] check_parent_health (log_config_mgr.cpp:2744) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=91] re_register_parent reason(palf_id=1002, self="10.0.0.3:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1713924044012381, last_first_register_time_us=1713875588469945, register_time_us=1713924044012381, curr_time_us=1713924048022513) [2024-04-24 10:00:48.022599] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201088}, multi_version_start=1713895479896246000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.022793] INFO [PALF] register_parent_ (log_config_mgr.cpp:2601) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=117] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="10.0.0.3:2882", reason=1, leader="10.0.0.2:2882", register_time_us=1713924048022649, last_submit_register_req_time_us=1713924048022649, register_parent_reason=1) [2024-04-24 10:00:48.022956] INFO [PALF] check_parent_health (log_config_mgr.cpp:2753) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=153] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="10.0.0.3:2882") [2024-04-24 10:00:48.023181] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201402}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.023747] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201196}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.024360] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200732}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.024980] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200706}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.025497] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200480}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.026202] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200364}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.026311] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=85][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.026737] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200746}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.027343] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201208}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.027942] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200264}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.028513] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201324}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.029189] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201350}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.029742] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200262}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.030412] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201112}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.030795] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [874434][T1004_PlanCache][T1004][Y0-0000000000000000-0-0] [lt=96] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1004) [2024-04-24 10:00:48.030957] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [874434][T1004_PlanCache][T1004][Y0-0000000000000000-0-0] [lt=154] start lib cache evict(tenant_id=1004, mem_hold=457179136, mem_limit=579820585, cache_obj_num=2606, cache_node_num=2600) [2024-04-24 10:00:48.031077] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [874434][T1004_PlanCache][T1004][Y0-0000000000000000-0-0] [lt=106] end lib cache evict(tenant_id=1004, cache_evict_num=0, mem_hold=457179136, mem_limit=579820585, cache_obj_num=2606, cache_node_num=2600) [2024-04-24 10:00:48.031095] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=125] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200764}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.031159] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [874434][T1004_PlanCache][T1004][Y0-0000000000000000-0-0] [lt=74] schedule next cache evict task(evict_interval=5000000) [2024-04-24 10:00:48.031682] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200500}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.032198] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201244}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.032699] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201366}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.033307] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200136}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.034024] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200950}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.034373] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=81] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-04-24 10:00:48.034662] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201398}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.035250] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201190}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.035808] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200914}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.036328] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=121] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200882}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.036954] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200310}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.037416] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200504}, multi_version_start=1713895480517479000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.038062] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200700}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.038251] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=134] no need to update min/max version(new_version={val:1713924045694942002, v:0}, min_version={val:1713924045694942002, v:0}, max_version={val:1713924045694942002, v:0}) [2024-04-24 10:00:48.038639] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200330}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.037484] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [874434][T1004_PlanCache][T1004][Y0-0000000000000000-0-0] [lt=94] Dumping All Cache Objs(alloc_obj_list.count()=2606, alloc_obj_list=[{obj_id:270, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173600}, {obj_id:271, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174960}, {obj_id:280, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:281, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:282, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:283, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:284, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:286, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:288, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:289, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:291, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:292, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:295, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:296, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:299, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:300, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178499}, {obj_id:301, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:302, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:303, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:304, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:307, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:309, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:310, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:311, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:313, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:314, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:315, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:316, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:317, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:318, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:319, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:320, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:321, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:322, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:323, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:324, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:325, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:326, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:327, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:328, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:329, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:330, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:331, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:332, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:333, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:334, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:335, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:336, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:337, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:338, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:339, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:340, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:341, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:342, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:343, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:344, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:345, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:346, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:347, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:348, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:349, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:350, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:351, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:352, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:353, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:354, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:355, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:356, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:357, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:358, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:359, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:360, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:361, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:362, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:363, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:364, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:365, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:366, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:367, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:368, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:369, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:370, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:371, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:372, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:373, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:374, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:375, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:376, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:381, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:382, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:383, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:384, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:385, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:386, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:387, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:388, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:389, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:390, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:391, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:392, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:393, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:394, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:395, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:396, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:397, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:398, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:399, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:400, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:401, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:402, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:403, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:404, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:405, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:406, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:407, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:408, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:409, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:410, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:411, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:412, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:413, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:415, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173296}, {obj_id:419, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:301071}, {obj_id:420, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246208}, {obj_id:421, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281861}, {obj_id:422, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268960}, {obj_id:423, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214161}, {obj_id:424, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253263}, {obj_id:425, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175408}, {obj_id:426, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178442}, {obj_id:427, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228175}, {obj_id:428, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186751}, {obj_id:429, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195314}, {obj_id:430, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173248}, {obj_id:431, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299295}, {obj_id:432, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232430}, {obj_id:433, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203884}, {obj_id:434, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210014}, {obj_id:435, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222957}, {obj_id:436, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:205775}, {obj_id:437, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:218381}, {obj_id:438, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197447}, {obj_id:439, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197445}, {obj_id:440, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191492}, {obj_id:441, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212278}, {obj_id:442, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197938}, {obj_id:445, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198158}, {obj_id:446, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188509}, {obj_id:447, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185401}, {obj_id:448, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184116}, {obj_id:449, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198513}, {obj_id:450, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191573}, {obj_id:451, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186879}, {obj_id:452, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184292}, {obj_id:453, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187220}, {obj_id:454, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184324}, {obj_id:455, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187221}, {obj_id:456, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184260}, {obj_id:457, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190471}, {obj_id:459, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185777}, {obj_id:460, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182076}, {obj_id:461, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208545}, {obj_id:462, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181884}, {obj_id:463, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186650}, {obj_id:464, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182284}, {obj_id:465, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190347}, {obj_id:466, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181980}, {obj_id:467, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183132}, {obj_id:468, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187473}, {obj_id:469, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175168}, {obj_id:470, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177704}, {obj_id:471, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202250}, {obj_id:472, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:305642}, {obj_id:473, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180374}, {obj_id:474, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:249607}, {obj_id:475, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184924}, {obj_id:476, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172304}, {obj_id:477, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:254861}, {obj_id:478, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:301987}, {obj_id:479, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245954}, {obj_id:482, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187993}, {obj_id:484, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226929}, {obj_id:485, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241889}, {obj_id:486, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220521}, {obj_id:487, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235257}, {obj_id:489, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185912}, {obj_id:490, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185912}, {obj_id:492, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182824}, {obj_id:493, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193849}, {obj_id:494, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187348}, {obj_id:495, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187372}, {obj_id:496, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:497, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173504}, {obj_id:498, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:499, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177273}, {obj_id:500, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:502, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181132}, {obj_id:503, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:504, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216328}, {obj_id:505, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179603}, {obj_id:506, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:507, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:509, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178892}, {obj_id:510, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174688}, {obj_id:512, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180165}, {obj_id:513, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186392}, {obj_id:514, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182904}, {obj_id:515, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180509}, {obj_id:516, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179088}, {obj_id:517, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180853}, {obj_id:518, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179119}, {obj_id:519, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180854}, {obj_id:520, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179062}, {obj_id:521, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182000}, {obj_id:522, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182404}, {obj_id:523, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179995}, {obj_id:524, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177793}, {obj_id:525, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191818}, {obj_id:526, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177809}, {obj_id:527, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180277}, {obj_id:528, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178217}, {obj_id:529, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182256}, {obj_id:530, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177905}, {obj_id:531, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178489}, {obj_id:532, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181655}, {obj_id:533, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:286440}, {obj_id:534, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:255974}, {obj_id:535, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:280815}, {obj_id:536, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:250354}, {obj_id:537, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180836}, {obj_id:538, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:539, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183476}, {obj_id:541, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187928}, {obj_id:542, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:543, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177119}, {obj_id:544, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:545, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179069}, {obj_id:546, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198078}, {obj_id:547, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172528}, {obj_id:548, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175256}, {obj_id:550, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217216}, {obj_id:551, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178297}, {obj_id:553, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182203}, {obj_id:555, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186953}, {obj_id:556, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180347}, {obj_id:558, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:559, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172080}, {obj_id:560, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191778}, {obj_id:561, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:562, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179119}, {obj_id:563, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187733}, {obj_id:564, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186020}, {obj_id:565, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:568, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178398}, {obj_id:569, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185485}, {obj_id:570, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:571, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194633}, {obj_id:572, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194633}, {obj_id:573, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183528}, {obj_id:574, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183528}, {obj_id:575, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172960}, {obj_id:576, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173696}, {obj_id:578, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:580, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174080}, {obj_id:583, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203274}, {obj_id:584, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180108}, {obj_id:585, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215880}, {obj_id:586, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:588, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203466}, {obj_id:590, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228543}, {obj_id:592, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:171872}, {obj_id:593, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175032}, {obj_id:594, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228351}, {obj_id:595, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184284}, {obj_id:596, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:598, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:599, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:600, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176736}, {obj_id:601, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:602, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175416}, {obj_id:603, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173104}, {obj_id:604, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:605, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210897}, {obj_id:606, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178793}, {obj_id:607, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:609, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289757}, {obj_id:612, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176820}, {obj_id:613, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:617, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178414}, {obj_id:618, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189901}, {obj_id:619, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272691}, {obj_id:620, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216666}, {obj_id:621, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:623, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181615}, {obj_id:624, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187140}, {obj_id:625, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:628, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:205860}, {obj_id:630, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198783}, {obj_id:631, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179440}, {obj_id:632, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212784}, {obj_id:633, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202696}, {obj_id:634, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199982}, {obj_id:635, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:636, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:639, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178921}, {obj_id:640, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185967}, {obj_id:641, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179568}, {obj_id:642, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:643, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:644, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:645, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:646, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:647, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:648, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:649, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:650, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:288992}, {obj_id:655, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160946}, {obj_id:658, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176907}, {obj_id:659, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176232}, {obj_id:660, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172912}, {obj_id:661, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185552}, {obj_id:662, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202696}, {obj_id:663, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199982}, {obj_id:664, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178921}, {obj_id:665, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185967}, {obj_id:666, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179568}, {obj_id:667, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:668, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:669, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:166013}, {obj_id:670, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:162314}, {obj_id:671, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157852}, {obj_id:672, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:673, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175542}, {obj_id:674, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157671}, {obj_id:677, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:678, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:679, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208603}, {obj_id:680, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186331}, {obj_id:681, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201118}, {obj_id:682, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186330}, {obj_id:683, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192344}, {obj_id:684, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177824}, {obj_id:685, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:686, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:687, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:688, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:689, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:690, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:691, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:692, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:693, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:694, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:695, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:696, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:697, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:698, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:699, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:700, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:702, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:703, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:705, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453841}, {obj_id:706, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174520}, {obj_id:708, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:709, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:710, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:711, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:712, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:713, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:714, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:715, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:716, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:717, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:718, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:719, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:720, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:721, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:722, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:723, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:724, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:725, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:726, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:727, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:728, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:729, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:730, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:731, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:732, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:733, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:734, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180314}, {obj_id:735, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182920}, {obj_id:736, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222053}, {obj_id:737, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:382675}, {obj_id:738, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203482}, {obj_id:739, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:740, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:741, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:742, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:743, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:744, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:745, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:746, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:747, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:748, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:749, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:750, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:751, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:752, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:753, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:754, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:755, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:756, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:757, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:758, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:759, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:760, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:761, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:762, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:763, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:764, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:765, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:766, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:767, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:768, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:769, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:770, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:771, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:772, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:773, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:774, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:775, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:776, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:777, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:778, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:779, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:780, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:781, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:782, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:783, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:784, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:785, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:786, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:787, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:788, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:789, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:790, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:791, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:792, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:793, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count: [2024-04-24 10:00:48.039128] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201086}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.039152] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [874434][T1004_PlanCache][T1004][Y0-0000000000000000-0-0] [lt=1655] schedule next cache evict task(evict_interval=5000000) [2024-04-24 10:00:48.039739] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201106}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.040314] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200448}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.040948] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200400}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.041378] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200994}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.041956] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200918}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.042548] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200178}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.043157] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201414}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.043515] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=81] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.043701] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200034}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.044276] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200664}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.044857] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200534}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.045403] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=119] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201262}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.045685] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AE-0-0] [lt=85][errcode=-4023] tx is 2pc logging(this={this:0x14b055e1a050, ref:3, trans_id:{txid:176904}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138442, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589701], ctx_create_time:1713924046585672}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046585674, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452001, v:0}, log_ts:{val:1713924046589452001, v:0}, lsn:{lsn:79580890}, submit_ts:1713924046589452}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176904}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176904}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452001, v:0}, start_scn:{val:1713924046589452001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877457407528 trans_version={val:1713924046589452001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176904} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3140423830 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:48.046002] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201326}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.045961] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AE-0-0] [lt=260][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e1a050, ref:3, trans_id:{txid:176904}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138442, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046589701], ctx_create_time:1713924046585672}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046585674, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046589452001, v:0}, log_ts:{val:1713924046589452001, v:0}, lsn:{lsn:79580890}, submit_ts:1713924046589452}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176904}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176904}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924046589452001, v:0}, start_scn:{val:1713924046589452001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877457407528 trans_version={val:1713924046589452001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176904} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3140423830 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046589452, block_frozen_memtable:null}) [2024-04-24 10:00:48.046091] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AE-0-0] [lt=122][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176904}) [2024-04-24 10:00:48.046164] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AE-0-0] [lt=63][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176904}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924048040436, cluster_id:1713874979}, expire_ts:1713925045138403, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:48.046252] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AE-0-0] [lt=65][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176904}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924048040436, cluster_id:1713874979}, expire_ts:1713925045138403, commit_start_scn:{val:1713924046579624000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:48.046511] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200122}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.047140] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201194}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.047563] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200184}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.048051] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201278}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.048443] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201258}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.049033] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200946}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.049482] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200168}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.049989] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201308}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.050463] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201388}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.050957] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200798}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.051461] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200270}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.051990] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201018}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.052045] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=85] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.052461] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201234}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.053064] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200718}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.053549] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200582}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.054049] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200021}, multi_version_start=1713895480517479001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.054540] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200748}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.055026] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200576}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.055566] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200612}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.056039] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200938}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.056213] INFO [STORAGE] exec_wait (ob_tmp_file_cache.cpp:1531) [1556243][T1004_ComTimerT][T1004][Y0-0000000000000000-0-0] [lt=88] succeed to do one round of tmp block io(ret=-4023, loop_nums=0, wait_io_cnt=0, washing_count=0, block_cache_num=0, page_cache_num=0) [2024-04-24 10:00:48.056565] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200598}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.057130] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=159] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201378}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.057599] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200618}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.058111] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200692}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.058540] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200654}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.059148] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200708}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.059555] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=55][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.059616] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200025}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.060145] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200204}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.060409] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [862260][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=134] no need to update min/max version(new_version={val:1713924045845177000, v:0}, min_version={val:1713924045845177000, v:0}, max_version={val:1713924045845177000, v:0}) [2024-04-24 10:00:48.060564] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:1152921504606846977}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.061082] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201482}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.061531] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200038}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.062015] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200528}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.062421] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201426}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.062938] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201442}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.063343] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200854}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.063844] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200464}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.064334] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=111] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200256}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.064832] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200696}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.065296] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201466}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.065848] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200714}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.066304] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=106] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200760}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.066491] INFO eloop_run (eloop.c:143) [861399][pnio1][T0][Y0-0000000000000000-0-0] [lt=125] PNIO [ratelimit] time: 1713924048066483, bytes: 14675767996, bw: 0.174281 MB/s, add_ts: 1007669, add_bytes: 184148 [2024-04-24 10:00:48.066739] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201248}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.067256] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200146}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.067684] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200346}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.068151] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200742}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.068540] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200996}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.069046] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201346}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.069554] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200740}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.070096] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201220}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.070493] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201082}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.071070] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201436}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.071500] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200354}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.072257] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=210] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.072357] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=88] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.072429] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=504] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200982}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.072451] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=88] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.072795] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200286}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.073295] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200756}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.073739] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200342}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.074269] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200452}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.074744] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200340}, multi_version_start=1713895480882383001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.075268] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201078}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.075691] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200456}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.076310] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201004}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.076796] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201152}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.077363] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=130] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201242}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.077824] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:1152921504606846980}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.078306] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200868}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.078759] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201186}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.079245] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200930}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.079741] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200712}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.080180] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200674}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.080683] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201476}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.081191] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200686}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.081443] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [874706][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=161] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:769, free_obj_hold_cnt:74, allocator used:3351768, allocator total:3466624}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:72, free_obj_hold_cnt:32, allocator used:6821464, allocator total:6821464}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():154752, full allocator used:1600, full allocator total:220288}, tablets_mem=10508376, tablets_mem_limit=257698020, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:23, free_obj_hold_cnt:163, allocator used:369024, allocator total:392448}, pending_cnt=0, wait_gc_count=0, tablet count=841) [2024-04-24 10:00:48.081714] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201236}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.082132] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200200}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.082593] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200460}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.083061] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200948}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.083478] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201178}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.084065] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200040}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.084470] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200392}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.085069] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201136}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.085474] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200922}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.086053] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=135] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200228}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.086480] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201202}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.087020] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200292}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.087450] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200792}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.087967] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201098}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.088383] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200454}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.088799] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201342}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.089263] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=53] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200020}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.089686] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=54] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200030}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.090216] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200045}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.090683] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201016}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.091185] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200864}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.091667] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201430}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.092165] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200526}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.092584] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201118}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.093218] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200942}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.093744] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200646}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.094226] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200296}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.094404] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C260A-0-0] [lt=69][errcode=-4023] tx is 2pc logging(this={this:0x14b15a16b090, ref:3, trans_id:{txid:391371}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924076773625, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046800195], ctx_create_time:1713924046784446}{ls_id:{id:1}, session_id:3221795340, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046791808, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046799990000, v:0}, log_ts:{val:1713924046799990000, v:0}, lsn:{lsn:314127056}, submit_ts:1713924046799990}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:391371}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:391371}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046799990000, v:0}, start_scn:{val:1713924046799990000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875442714112 max_table_version=1713875442714112 trans_version={val:1713924046799990000, v:0} commit_version={val:1713924046799990000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:391371} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=2612980561 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046799990000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046799990, block_frozen_memtable:null}) [2024-04-24 10:00:48.094698] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200039}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.094637] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C260A-0-0] [lt=216][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b15a16b090, ref:3, trans_id:{txid:391371}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924076773625, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046800195], ctx_create_time:1713924046784446}{ls_id:{id:1}, session_id:3221795340, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046791808, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046799990000, v:0}, log_ts:{val:1713924046799990000, v:0}, lsn:{lsn:314127056}, submit_ts:1713924046799990}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:391371}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:391371}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046799990000, v:0}, start_scn:{val:1713924046799990000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875442714112 max_table_version=1713875442714112 trans_version={val:1713924046799990000, v:0} commit_version={val:1713924046799990000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:391371} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=2612980561 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046799990000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046799990, block_frozen_memtable:null}) [2024-04-24 10:00:48.094798] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C260A-0-0] [lt=153][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:391371}) [2024-04-24 10:00:48.094976] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C260A-0-0] [lt=171][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1001, tx_id:{txid:391371}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924048089689, cluster_id:1713874979}, expire_ts:1713924076795547, commit_start_scn:{val:1713924046786976000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:48.095095] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2B27C260A-0-0] [lt=106][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1001, tx_id:{txid:391371}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924048089689, cluster_id:1713874979}, expire_ts:1713924076795547, commit_start_scn:{val:1713924046786976000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:48.095315] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200066}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.095823] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200910}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.096339] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200350}, multi_version_start=1713895481019382000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.096835] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200772}, multi_version_start=1713895481263061000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.097335] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201456}, multi_version_start=1713895481263061000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.097834] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200210}, multi_version_start=1713895481263061000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.098317] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200037}, multi_version_start=1713895481263061000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.098851] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=121] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200986}, multi_version_start=1713895481263061000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.099312] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200890}, multi_version_start=1713895481263061000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.099835] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201406}, multi_version_start=1713895481263061000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.100311] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=119] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200406}, multi_version_start=1713895481263061000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.100824] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201320}, multi_version_start=1713895481263061000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.101281] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201298}, multi_version_start=1713895481263061000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.101271] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=73] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.101358] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=72] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.101456] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=90] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.101790] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200462}, multi_version_start=1713895481263061000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.102240] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=120] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201306}, multi_version_start=1713895481263061000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.102706] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200302}, multi_version_start=1713895481263061000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.103160] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201412}, multi_version_start=1713895481263061000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.103729] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200614}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.104507] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=353] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201460}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.105170] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200132}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.105638] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200644}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.106306] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200486}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.106788] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200348}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.107341] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=164] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201154}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.107826] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200794}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.108244] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=121] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8886493184, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:48.108391] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=172] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201420}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.108826] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200806}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.109309] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200588}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.109800] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200776}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.110300] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [861686][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=75] get_max_applied_scn(scn={val:1, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1, v:0}}) [2024-04-24 10:00:48.110396] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=142] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200378}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.110438] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [861686][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=120] get_min_unreplayed_log_info(lsn={lsn:179739101}, scn={val:1713924046890900001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713924047752743, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:18156, ret_code_:0}}, next_to_submit_lsn_:{lsn:179739101}, next_to_submit_scn_:{val:1713924046890900001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x14b11a8c9000, next_round_pread_size:3678208, curr_read_pos:2869, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2869, log_storage_:{IteratorStorage:{start_lsn:{lsn:179736232}, end_lsn:{lsn:179739101}, read_buf:{buf_len_:389120, buf_:0x14b11a8c9000}, block_size:67104768, log_storage_:0x14b186b4d470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713924046890900000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:677, scn_:{val:1713924046890900000, v:0}, data_checksum:2466452222, flag:0}}, init_mode_version:0, accumulate_checksum:2186446675, curr_entry_is_padding:0, padding_entry_size:709, padding_entry_scn:{val:1713924046890900000, v:0}}}}}) [2024-04-24 10:00:48.110998] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=123] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201054}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.111490] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200284}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.111985] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201282}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.112495] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200920}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.113326] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=250] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200702}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.113855] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200220}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.114493] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=161] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201374}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.115113] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=122] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201252}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.115635] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201092}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.116204] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201006}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.116777] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200648}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.117269] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201218}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.117812] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200196}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.118327] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=130] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200494}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.118924] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201130}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.119331] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201150}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.119984] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201280}, multi_version_start=1713895481376608000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.120417] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201386}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.120982] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200154}, multi_version_start=1713895481443418000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.121443] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200120}, multi_version_start=1713895481443418000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.122052] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201352}, multi_version_start=1713895481443418000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.122512] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200896}, multi_version_start=1713895481443418000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.123076] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200600}, multi_version_start=1713895481443418000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.123594] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200250}, multi_version_start=1713895481443418000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.124241] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=109] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201292}, multi_version_start=1713895481443418000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.124769] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201396}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.125322] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200666}, multi_version_start=1713895481443418000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.125832] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200562}, multi_version_start=1713895481443418000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.126396] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=78][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.126509] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=213] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201100}, multi_version_start=1713895481443418000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.127138] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201444}, multi_version_start=1713895481443418000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.127672] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200316}, multi_version_start=1713895481443418000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.128233] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201104}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.128762] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200142}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.129320] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200216}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.129793] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201380}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.130310] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=129] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201266}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.130763] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201040}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.131337] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200068}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.131952] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200074}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.132619] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=115] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201222}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.133078] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200572}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.133599] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200926}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.134171] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200978}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.134772] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200112}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.135307] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=167] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200694}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.135905] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200892}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.136487] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=122] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201020}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.137085] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201448}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.137795] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200294}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.138529] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=249] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200888}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.139412] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=134] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200290}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.140024] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=109] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201376}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.140561] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201184}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.141251] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=121] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201000}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.141507] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [874742][T1003_LSArchive][T1003][YB420A000003-000616C2B608F0E7-0-0] [lt=66] gc stale ls task succ [2024-04-24 10:00:48.142012] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200036}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.142376] INFO [PALF] check_parent_health (log_config_mgr.cpp:2744) [861573][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=87] re_register_parent reason(palf_id=1, self="10.0.0.3:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1713924044133207, last_first_register_time_us=1713875184097509, register_time_us=1713924044133207, curr_time_us=1713924048142367) [2024-04-24 10:00:48.142588] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201002}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.142650] INFO [PALF] register_parent_ (log_config_mgr.cpp:2601) [861573][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=133] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1, self="10.0.0.3:2882", reason=3, leader="10.0.0.2:2882", register_time_us=1713924048142522, last_submit_register_req_time_us=1713924048142522, register_parent_reason=3) [2024-04-24 10:00:48.142739] INFO [PALF] check_parent_health (log_config_mgr.cpp:2753) [861573][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=82] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="10.0.0.3:2882") [2024-04-24 10:00:48.143240] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201438}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.143829] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=91] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.143855] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201034}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.144441] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=138] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201224}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.144719] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262C-0-0] [lt=121][errcode=-4023] tx is 2pc logging(this={this:0x14b0fbedf050, ref:3, trans_id:{txid:338711}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924076767546, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046792910], ctx_create_time:1713924046778243}{ls_id:{id:1}, session_id:3221795357, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046789276, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046791807000, v:0}, log_ts:{val:1713924046791807000, v:0}, lsn:{lsn:298513878}, submit_ts:1713924046792973}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:338711}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338711}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046791807000, v:0}, start_scn:{val:1713924046791807000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924046791807000, v:0} commit_version={val:1713924046791807000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338711} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=215183838 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046791807000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046792973, block_frozen_memtable:null}) [2024-04-24 10:00:48.144975] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262C-0-0] [lt=249][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fbedf050, ref:3, trans_id:{txid:338711}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924076767546, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046792910], ctx_create_time:1713924046778243}{ls_id:{id:1}, session_id:3221795357, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046789276, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046791807000, v:0}, log_ts:{val:1713924046791807000, v:0}, lsn:{lsn:298513878}, submit_ts:1713924046792973}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:338711}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338711}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046791807000, v:0}, start_scn:{val:1713924046791807000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924046791807000, v:0} commit_version={val:1713924046791807000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338711} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=215183838 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046791807000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046792973, block_frozen_memtable:null}) [2024-04-24 10:00:48.145117] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201060}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.145136] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262C-0-0] [lt=149][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338711}) [2024-04-24 10:00:48.145216] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262C-0-0] [lt=69][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338711}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924048140136, cluster_id:1713874979}, expire_ts:1713924076788074, commit_start_scn:{val:1713924046780604000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:48.145342] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262C-0-0] [lt=103][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338711}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924048140136, cluster_id:1713874979}, expire_ts:1713924076788074, commit_start_scn:{val:1713924046780604000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:48.145354] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AF-0-0] [lt=106][errcode=-4023] tx is 2pc logging(this={this:0x14b147fffbd0, ref:4, trans_id:{txid:176906}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925046666492, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046708409], ctx_create_time:1713924046701144}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79581303}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046705015, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046707343000, v:0}, log_ts:{val:1713924046707343000, v:0}, lsn:{lsn:79582268}, submit_ts:1713924046708648}, this:0x14b148002b40, is_inited_:true, trans_id:{txid:176906}, ls_id:{id:1}, ctx:0x14b147fffbd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176906}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046707343000, v:0}, start_scn:{val:1713924046589452002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877459227536 max_table_version=1713877465782000 trans_version={val:1713924046707343000, v:0} commit_version={val:1713924046707343000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176906} ls_id=1 callback_alloc_count=5 callback_free_count=0 checksum=3165083175 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=8 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=8 unsynced_cnt=5 unsubmitted_cnt_=0 cb_statistics:[main=8, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046708648, block_frozen_memtable:null}) [2024-04-24 10:00:48.145656] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200592}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.145550] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AF-0-0] [lt=188][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b147fffbd0, ref:4, trans_id:{txid:176906}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925046666492, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046708409], ctx_create_time:1713924046701144}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79581303}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046705015, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046707343000, v:0}, log_ts:{val:1713924046707343000, v:0}, lsn:{lsn:79582268}, submit_ts:1713924046708648}, this:0x14b148002b40, is_inited_:true, trans_id:{txid:176906}, ls_id:{id:1}, ctx:0x14b147fffbd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176906}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046707343000, v:0}, start_scn:{val:1713924046589452002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877459227536 max_table_version=1713877465782000 trans_version={val:1713924046707343000, v:0} commit_version={val:1713924046707343000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176906} ls_id=1 callback_alloc_count=5 callback_free_count=0 checksum=3165083175 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=8 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=8 unsynced_cnt=5 unsubmitted_cnt_=0 cb_statistics:[main=8, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046589452002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046708648, block_frozen_memtable:null}) [2024-04-24 10:00:48.145731] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AF-0-0] [lt=174][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176906}) [2024-04-24 10:00:48.145819] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AF-0-0] [lt=73][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176906}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924048140617, cluster_id:1713874979}, expire_ts:1713925046666035, commit_start_scn:{val:1713924046703599000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:48.145991] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04AF-0-0] [lt=157][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176906}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924048140617, cluster_id:1713874979}, expire_ts:1713925046666035, commit_start_scn:{val:1713924046703599000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:48.146201] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200128}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.146751] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201048}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.147345] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200638}, multi_version_start=1713895481589823000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.147854] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201458}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.148441] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200338}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.149085] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200508}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.149646] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200388}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.150215] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201294}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.150657] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200182}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.151239] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200444}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.151730] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201452}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.152415] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=92] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.152450] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200024}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.153063] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=118] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201170}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.153582] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200476}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.154111] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201030}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.154591] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200376}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.155193] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200836}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.155825] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200578}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.155825] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E97-0-0] [lt=72][errcode=-4023] tx is 2pc logging(this={this:0x14b168f2d710, ref:3, trans_id:{txid:877456}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924646597291, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046606334], ctx_create_time:1713924046597291}{ls_id:{id:1001}, session_id:3221852891, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046603230, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046605790000, v:0}, log_ts:{val:1713924046605790000, v:0}, lsn:{lsn:1002185585}, submit_ts:1713924046605790}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877456}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877456}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046605790000, v:0}, start_scn:{val:1713924046605790000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924046605790000, v:0} commit_version={val:1713924046605790000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877456} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=737308951 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046605790000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046605790, block_frozen_memtable:null}) [2024-04-24 10:00:48.156055] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E97-0-0] [lt=220][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b168f2d710, ref:3, trans_id:{txid:877456}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924646597291, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924046606334], ctx_create_time:1713924046597291}{ls_id:{id:1001}, session_id:3221852891, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924046603230, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924046605790000, v:0}, log_ts:{val:1713924046605790000, v:0}, lsn:{lsn:1002185585}, submit_ts:1713924046605790}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877456}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877456}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924046605790000, v:0}, start_scn:{val:1713924046605790000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924046605790000, v:0} commit_version={val:1713924046605790000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877456} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=737308951 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924046605790000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924046605790, block_frozen_memtable:null}) [2024-04-24 10:00:48.156201] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E97-0-0] [lt=138][errcode=-4023] commit fail(ret=-4023, coord={id:1001}, tx_id={txid:877456}) [2024-04-24 10:00:48.156279] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E97-0-0] [lt=72][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877456}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924048151155, cluster_id:1713874979}, expire_ts:1713924558320000, commit_start_scn:{val:1713924046594982000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:48.156381] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=126] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201290}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.156406] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E97-0-0] [lt=102][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877456}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924048151155, cluster_id:1713874979}, expire_ts:1713924558320000, commit_start_scn:{val:1713924046594982000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:48.156971] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201064}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.157576] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200730}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.158181] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=137] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200208}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.158696] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200308}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.159252] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201094}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.159727] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=121][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.159827] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200602}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.160340] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=134] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200520}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.160848] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200972}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.161479] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=146] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201428}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.162076] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=107] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200194}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.162600] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200628}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.163161] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=118] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200118}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.163680] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200244}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.164340] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201014}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.164945] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200564}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.165441] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200744}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.166096] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200538}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.166524] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=54] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200138}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.167123] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200992}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.167641] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201062}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.168162] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200898}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.168676] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201134}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.169212] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200484}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.169769] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200160}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.170374] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200552}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.170851] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201162}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.171403] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=159] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200734}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.171932] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200788}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.172402] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=52] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.172449] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=118] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200478}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.172503] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=87] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.172617] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=106] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.173088] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200872}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.173688] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200762}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.174246] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201044}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.174963] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=118] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200468}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.175500] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201416}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.176120] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200258}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.176206] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B4AC6E62-0-0] [lt=91][errcode=-4023] tx is 2pc logging(this={this:0x14b168f30dd0, ref:3, trans_id:{txid:877457}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924647763173, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924047778260], ctx_create_time:1713924047763173}{ls_id:{id:1001}, session_id:3221998686, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924047770485, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924047777372000, v:0}, log_ts:{val:1713924047777372000, v:0}, lsn:{lsn:1002187270}, submit_ts:1713924047777372}, this:0x14b168f33d40, is_inited_:true, trans_id:{txid:877457}, ls_id:{id:1001}, ctx:0x14b168f30dd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877457}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924047777372000, v:0}, start_scn:{val:1713924047777372000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924047777372000, v:0} commit_version={val:1713924047777372000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877457} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=106315759 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924047777372000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924047778783, block_frozen_memtable:null}) [2024-04-24 10:00:48.176361] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B4AC6E62-0-0] [lt=146][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b168f30dd0, ref:3, trans_id:{txid:877457}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924647763173, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924047778260], ctx_create_time:1713924047763173}{ls_id:{id:1001}, session_id:3221998686, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924047770485, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924047777372000, v:0}, log_ts:{val:1713924047777372000, v:0}, lsn:{lsn:1002187270}, submit_ts:1713924047777372}, this:0x14b168f33d40, is_inited_:true, trans_id:{txid:877457}, ls_id:{id:1001}, ctx:0x14b168f30dd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877457}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924047777372000, v:0}, start_scn:{val:1713924047777372000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924047777372000, v:0} commit_version={val:1713924047777372000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877457} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=106315759 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924047777372000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924047778783, block_frozen_memtable:null}) [2024-04-24 10:00:48.176477] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B4AC6E62-0-0] [lt=106][errcode=-4023] commit fail(ret=-4023, coord={id:1001}, tx_id={txid:877457}) [2024-04-24 10:00:48.176535] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B4AC6E62-0-0] [lt=47][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877457}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924048171473, cluster_id:1713874979}, expire_ts:1713924618349599, commit_start_scn:{val:1713924047757555000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:48.176622] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201272}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.176639] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B4AC6E62-0-0] [lt=79][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877457}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924048171473, cluster_id:1713874979}, expire_ts:1713924618349599, commit_start_scn:{val:1713924047757555000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:48.177141] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200506}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.177668] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200394}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.178226] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200272}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.178672] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200192}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.179206] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200382}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.179859] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200956}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.180583] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=157] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200274}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.181114] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201156}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.181611] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2405) [2325190][T1002_L0_G19][T1002][YB420A000002-000616C2B36C4C47-0-0] [lt=96] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"10.0.0.2:2882", ls_array_:[{id:1}, {id:1001}]}) [2024-04-24 10:00:48.181668] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200472}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.181771] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1509) [2325190][T1002_L0_G19][T1002][YB420A000002-000616C2B36C4C47-0-0] [lt=117] get_member_gc_stat(is_valid_member=true, stat=1, member_list=2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="10.0.0.2:2882", this={role:1, proposal_id:5, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2024-04-24 10:00:48.181941] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1509) [2325190][T1002_L0_G19][T1002][YB420A000002-000616C2B36C4C47-0-0] [lt=89] get_member_gc_stat(is_valid_member=true, stat=1, member_list=2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="10.0.0.2:2882", this={role:1, proposal_id:7, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2024-04-24 10:00:48.182226] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200550}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.182728] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200850}, multi_version_start=1713895481731612000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.183252] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200246}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.183785] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201300}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.184286] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201246}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.185337] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200070}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.186036] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201344}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.186615] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=98] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201360}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.187309] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=129] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201160}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.187994] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=104] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201166}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.188604] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201050}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.189282] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200962}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.189910] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=98] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200684}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.190519] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200838}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.191188] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200542}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.191676] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201168}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.192310] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=104] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201140}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.192782] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201180}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.193421] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=147] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201302}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.194024] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200604}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.194502] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201454}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.195021] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201470}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.195522] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200640}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.195862] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [862434][T1001_CKClogDis][T1001][Y0-0000000000000000-0-0] [lt=159] ====== check clog disk timer task ====== [2024-04-24 10:00:48.195999] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [862434][T1001_CKClogDis][T1001][Y0-0000000000000000-0-0] [lt=119] get_disk_usage(ret=0, capacity(MB):=614, used(MB):=363) [2024-04-24 10:00:48.196047] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201256}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.196199] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [862434][T1001_CKClogDis][T1001][Y0-0000000000000000-0-0] [lt=122] cannot_recycle_log_size statistics(cannot_recycle_log_size=179917398, threshold=193273528, need_update_checkpoint_scn=false) [2024-04-24 10:00:48.196591] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200862}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.197096] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200704}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.197669] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200758}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.198174] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200076}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.198695] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:1152921504606846985}, multi_version_start=1713899298357631000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.199210] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201322}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.199715] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200934}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.200119] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2405) [2325200][T1003_L0_G19][T1003][YB420A000002-000616C2B89C25AF-0-0] [lt=165] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1003, self_addr_:"10.0.0.2:2882", ls_array_:[{id:1}]}) [2024-04-24 10:00:48.200263] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201484}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.200227] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1509) [2325200][T1003_L0_G19][T1003][YB420A000002-000616C2B89C25AF-0-0] [lt=86] get_member_gc_stat(is_valid_member=true, stat=1, member_list=2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="10.0.0.2:2882", this={role:1, proposal_id:7, palf_env_:0x14b14219e030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2024-04-24 10:00:48.200964] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:1152921504606846978}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.201471] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200584}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.201491] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=60] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.201572] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=61] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.201666] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=88] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.202085] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201172}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.202526] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200176}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.203058] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200043}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.203562] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200078}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.204192] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201410}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.204665] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200186}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.205248] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200150}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.205991] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201108}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.206543] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200856}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.207125] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200282}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.207548] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200988}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.208106] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200540}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.208600] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200318}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.209076] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201312}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.209753] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=128] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200906}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.210422] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201364}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.210941] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200238}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.211447] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200502}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.211987] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200482}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.212503] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200936}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.213030] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201284}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.213529] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201400}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.214046] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:1152921504606846993}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.214589] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201432}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.215127] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200442}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.215644] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200632}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.216224] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201332}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.216433] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F6D-0-0] [lt=97][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="10.0.0.3:2882", server="10.0.0.2:2882", update_func={old_end_lsn:{lsn:298513336}, new_end_lsn:{lsn:298514948}, old_advance_time_us:1713924046744985, new_ack_time_us:1713924048216410, advance delay(us):1471425}) [2024-04-24 10:00:48.216700] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F6D-0-0] [lt=96][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=482543, log_task={header:{begin_lsn:{lsn:298513336}, end_lsn:{lsn:298514948}, log_id:482543, min_scn:{val:1713924046684930000, v:0}, max_scn:{val:1713924046848982000, v:0}, data_len:1556, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:298513336}, data_checksum:3423088645, accum_checksum:3897560740, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924046685067, freeze_ts:1713924047734431, submit_ts:1713924047734608, flushed_ts:1713924047745220, gen_to_freeze cost time:1049364, gen_to_submit cost time:1049541, submit_to_flush cost time:10612}, fs_cb_begin_ts=1713924048216568, log_life_time=1531501) [2024-04-24 10:00:48.216934] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201146}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.216848] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=91][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924046684930000, v:0}, log_ts:{val:1713924046684930000, v:0}, lsn:{lsn:298513392}, submit_ts:1713924046684930}, this:0x14b0fbee5680, is_inited_:true, trans_id:{txid:548716}, ls_id:{id:1}, ctx:0x14b0fbee2710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1531889, ctx_lock_wait_time=17) [2024-04-24 10:00:48.217275] WDIAG [CLOG] statistics_cb_cost_ (ob_log_apply_service.cpp:933) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=234][errcode=-4389] cb cost too much time(lsn={lsn:298513392}, scn={val:1713924046684930000, v:0}, idx=16, total_cost_time=1532135, append_cost_time=50, cb_wait_thread_time=1531715, cb_wait_commit_time=0, cb_cost_time=370, append_start_time=1713924046685045, append_finish_time=1713924046685095, cb_first_handle_time=1713924048216810, cb_finish_time=1713924048217180) [2024-04-24 10:00:48.217422] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=86][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924046848982000, v:0}, log_ts:{val:1713924046848982000, v:0}, lsn:{lsn:298514462}, submit_ts:1713924046848982}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:548717}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1368410, ctx_lock_wait_time=17) [2024-04-24 10:00:48.217470] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200248}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.217696] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=83][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924046791807000, v:0}, log_ts:{val:1713924046791807000, v:0}, lsn:{lsn:298513878}, submit_ts:1713924046792973}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:338711}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1424702, ctx_lock_wait_time=12) [2024-04-24 10:00:48.218042] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=84] current clock is (clock_=0, max_seq_=0, mem_limit=264912214, cur_mem_hold=121634816, attr_.tenant_id_=1003) [2024-04-24 10:00:48.218129] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200320}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.218319] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=102] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2024-04-24 10:00:48.218632] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201068}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.219201] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200072}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.219530] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=79] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1003, server_version={version:{val:1713924045997782000, v:0}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=2221732, in_cluster_service=true, cluster_version={val:1713924045997782001, v:0}, min_cluster_version={val:1713924045997782001, v:0}, max_cluster_version={val:1713924045997782001, v:0}, get_cluster_version_err=0, cluster_version_delta=2221732, cluster_service_master="10.0.0.3:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=332207, succ_cluster_heartbeat_count=254, cluster_heartbeat_interval=50000, local_cluster_version={val:1713923467504080000, v:0}, local_cluster_delta=580715434, force_self_check=false, weak_read_refresh_interval=100000) [2024-04-24 10:00:48.219721] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200860}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.220324] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=109] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200848}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.220351] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [874754][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=63] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:48.220949] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200164}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.221093] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861496][RpcIO][T0][Y0-0000000000000000-0-0] [lt=86] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:48.221187] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861495][RpcIO][T0][Y0-0000000000000000-0-0] [lt=59] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:48.221478] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201316}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.221983] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861497][RpcIO][T0][Y0-0000000000000000-0-0] [lt=92] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:48.221983] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861498][RpcIO][T0][Y0-0000000000000000-0-0] [lt=76] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:48.222039] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200878}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.222458] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200852}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.222993] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200234}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.223402] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200964}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.223947] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201372}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.224386] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200974}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.224962] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200832}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.225405] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201116}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.225965] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200356}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.226424] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200626}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.226581] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=79] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1004, epoch:-1, self:"10.0.0.3:2882", switch_to_leader_ts:-1}) [2024-04-24 10:00:48.226708] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=114] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1004, epoch:-1, self:"10.0.0.3:2882", switch_to_leader_ts:-1}) [2024-04-24 10:00:48.226785] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=71][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.227081] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=119] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200174}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.227601] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200668}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.228228] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=129] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200830}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.228698] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200254}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.229274] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200368}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.229825] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200596}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.230416] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=109] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201338}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.230906] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200594}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.231470] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200778}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.231979] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201354}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.232438] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201012}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.233036] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200408}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.233552] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200490}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.234064] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200360}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.234586] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200530}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.234740] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=106] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-04-24 10:00:48.235145] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201240}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.235605] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201474}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.236114] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200812}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.236677] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201254}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.237247] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200546}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.237798] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201480}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.238330] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200554}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.238723] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=84] no need to update min/max version(new_version={val:1713924045694942002, v:0}, min_version={val:1713924045694942002, v:0}, max_version={val:1713924045694942002, v:0}) [2024-04-24 10:00:48.238983] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201102}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.239411] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:1152921504606847003}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.240076] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200362}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.240662] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200658}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.241174] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200488}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.241758] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201424}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.242302] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=150] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201198}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.243003] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201230}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.243536] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200288}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.244080] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD27B-0-0] [lt=78][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="10.0.0.3:2882", server="10.0.0.2:2882", update_func={old_end_lsn:{lsn:79580834}, new_end_lsn:{lsn:79585532}, old_advance_time_us:1713924046763311, new_ack_time_us:1713924048244072, advance delay(us):1480761}) [2024-04-24 10:00:48.244191] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=129] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201056}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.244295] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD1417-0-0] [lt=93][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=364235, log_task={header:{begin_lsn:{lsn:179739101}, end_lsn:{lsn:179740129}, log_id:364235, min_scn:{val:1713924047744490001, v:0}, max_scn:{val:1713924047744490001, v:0}, data_len:972, proposal_id:5, prev_lsn:{lsn:179738336}, prev_proposal_id:5, committed_end_lsn:{lsn:179739101}, data_checksum:591537631, accum_checksum:3255040915, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924047759592, freeze_ts:1713924047759600, submit_ts:1713924047759958, flushed_ts:1713924047768211, gen_to_freeze cost time:8, gen_to_submit cost time:366, submit_to_flush cost time:8253}, fs_cb_begin_ts=1713924048244242, log_life_time=484650) [2024-04-24 10:00:48.244287] INFO [SHARE] get_next_sess_id (ob_active_session_guard.cpp:360) [2326860][][T0][Y0-0000000000000000-0-0] [lt=0] succ to generate background session id(sessid=42923903156224) [2024-04-24 10:00:48.244280] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=117] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.244392] INFO register_pm (ob_page_manager.cpp:40) [2326860][][T0][Y0-0000000000000000-0-0] [lt=93] register pm finish(ret=0, &pm=0x14b10e8d4d40, pm.get_tid()=2326860, tenant_id=500) [2024-04-24 10:00:48.244360] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD27B-0-0] [lt=181][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=304850, log_task={header:{begin_lsn:{lsn:79580834}, end_lsn:{lsn:79585532}, log_id:304850, min_scn:{val:1713924046589452001, v:0}, max_scn:{val:1713924046872416000, v:0}, data_len:4642, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:79580834}, data_checksum:2935867450, accum_checksum:1233917234, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924046589951, freeze_ts:1713924047743606, submit_ts:1713924047744048, flushed_ts:1713924047760355, gen_to_freeze cost time:1153655, gen_to_submit cost time:1154097, submit_to_flush cost time:16307}, fs_cb_begin_ts=1713924048244304, log_life_time=1654353) [2024-04-24 10:00:48.244689] INFO [SHARE] pre_run (ob_tenant_base.cpp:314) [2326860][][T1001][Y0-0000000000000000-0-0] [lt=58] tenant thread pre_run(MTL_ID()=1001, ret=0, thread_count_=166) [2024-04-24 10:00:48.244681] INFO [SERVER.OMT] recv_group_request (ob_tenant.cpp:1292) [861399][pnio1][T0][YB420A000002-000616C2BD9FB8F8-0-0] [lt=63] worker thread created(id()=1001, group->group_id_=1) [2024-04-24 10:00:48.244837] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [2326860][][T1001][Y0-0000000000000000-0-0] [lt=142] Init thread local success [2024-04-24 10:00:48.244831] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200035}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.244857] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C39-0-0] [lt=67][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1001, self="10.0.0.3:2882", server="10.0.0.2:2882", update_func={old_end_lsn:{lsn:1002185529}, new_end_lsn:{lsn:1002187102}, old_advance_time_us:1713924046752336, new_ack_time_us:1713924048244851, advance delay(us):1492515}) [2024-04-24 10:00:48.244972] INFO unregister_pm (ob_page_manager.cpp:50) [2326860][][T1001][Y0-0000000000000000-0-0] [lt=112] unregister pm finish(&pm=0x14b10e8d4d40, pm.get_tid()=2326860) [2024-04-24 10:00:48.245043] INFO register_pm (ob_page_manager.cpp:40) [2326860][][T1001][Y0-0000000000000000-0-0] [lt=61] register pm finish(ret=0, &pm=0x14b10e8d4d40, pm.get_tid()=2326860, tenant_id=1001) [2024-04-24 10:00:48.245049] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C39-0-0] [lt=132][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=673414, log_task={header:{begin_lsn:{lsn:1002185529}, end_lsn:{lsn:1002187102}, log_id:673414, min_scn:{val:1713924046605790000, v:0}, max_scn:{val:1713924046605790000, v:0}, data_len:1517, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1002185529}, data_checksum:1894035115, accum_checksum:978172226, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924046606507, freeze_ts:1713924047734751, submit_ts:1713924047735075, flushed_ts:1713924047750261, gen_to_freeze cost time:1128244, gen_to_submit cost time:1128568, submit_to_flush cost time:15186}, fs_cb_begin_ts=1713924048245001, log_life_time=1638494) [2024-04-24 10:00:48.245033] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD27C-0-0] [lt=174][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=299912, log_task={header:{begin_lsn:{lsn:40356763}, end_lsn:{lsn:40357212}, log_id:299912, min_scn:{val:1713924047781305000, v:0}, max_scn:{val:1713924047781305000, v:0}, data_len:393, proposal_id:5, prev_lsn:{lsn:40356249}, prev_proposal_id:5, committed_end_lsn:{lsn:40356763}, data_checksum:4020602225, accum_checksum:2596594815, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924047783445, freeze_ts:1713924047783453, submit_ts:1713924047783685, flushed_ts:1713924047793777, gen_to_freeze cost time:8, gen_to_submit cost time:240, submit_to_flush cost time:10092}, fs_cb_begin_ts=1713924048244982, log_life_time=461537) [2024-04-24 10:00:48.245067] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [861712][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=81] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=608, total_release_part_ctx_count=199) [2024-04-24 10:00:48.245356] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C3A-0-0] [lt=85][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="10.0.0.3:2882", server="10.0.0.2:2882", update_func={old_end_lsn:{lsn:67158729}, new_end_lsn:{lsn:67158851}, old_advance_time_us:1713924047127140, new_ack_time_us:1713924048245346, advance delay(us):1118206}) [2024-04-24 10:00:48.245379] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB8F8-0-0] [lt=72] init point thread id with(&point=0x55f31f7e6c40, idx_=332, point=[thread id=2326860, timeout ts=08:00:00.0, last click point="(null):(null):0", last click ts=08:00:00.0], thread_id=2326860) [2024-04-24 10:00:48.245389] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=118] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200634}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.245430] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C3A-0-0] [lt=67] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=300) [2024-04-24 10:00:48.245430] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F27D-0-0] [lt=105][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924046872416000, v:0}, log_ts:{val:1713924046872416000, v:0}, lsn:{lsn:79584822}, submit_ts:1713924046872416}, this:0x14b14807de40, is_inited_:true, trans_id:{txid:327174}, ls_id:{id:1}, ctx:0x14b14807aed0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1373000, ctx_lock_wait_time=7) [2024-04-24 10:00:48.245491] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB8F8-0-0] [lt=82][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="10.0.0.3:2882", server="10.0.0.2:2882", update_func={old_end_lsn:{lsn:314126934}, new_end_lsn:{lsn:314128793}, old_advance_time_us:1713924046751193, new_ack_time_us:1713924048245481, advance delay(us):1494288}) [2024-04-24 10:00:48.245491] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874554][T1004_ApplySrv2][T1004][YB420A000003-000616C2BFC8F285-0-0] [lt=79][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924046707343000, v:0}, log_ts:{val:1713924046707343000, v:0}, lsn:{lsn:79582268}, submit_ts:1713924046708648}, this:0x14b148002b40, is_inited_:true, trans_id:{txid:176906}, ls_id:{id:1}, ctx:0x14b147fffbd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1536820, ctx_lock_wait_time=13) [2024-04-24 10:00:48.245636] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB8F8-0-0] [lt=64][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=505455, log_task={header:{begin_lsn:{lsn:314126934}, end_lsn:{lsn:314128793}, log_id:505455, min_scn:{val:1713924046407795000, v:0}, max_scn:{val:1713924046942380000, v:0}, data_len:1803, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:314126934}, data_checksum:4145299841, accum_checksum:1356787447, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924046622441, freeze_ts:1713924047734847, submit_ts:1713924047735138, flushed_ts:1713924047750057, gen_to_freeze cost time:1112406, gen_to_submit cost time:1112697, submit_to_flush cost time:14919}, fs_cb_begin_ts=1713924048245566, log_life_time=1623125) [2024-04-24 10:00:48.245645] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C3A-0-0] [lt=81][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=259313, log_task={header:{begin_lsn:{lsn:67158729}, end_lsn:{lsn:67158851}, log_id:259313, min_scn:{val:1713924047777372000, v:0}, max_scn:{val:1713924047777372000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:67158729}, data_checksum:667309872, accum_checksum:2601248750, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924047800749, freeze_ts:1713924047800758, submit_ts:1713924047800935, flushed_ts:1713924047813012, gen_to_freeze cost time:9, gen_to_submit cost time:186, submit_to_flush cost time:12077}, fs_cb_begin_ts=1713924048245577, log_life_time=444828) [2024-04-24 10:00:48.245802] INFO [LIB] stat (utility.h:1197) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F27D-0-0] [lt=97] [APPLY STAT CB APPEND COST TIME](cur_stat_count=31, stat_interval=5000000, avg (count/cost)=141, this=0x14b14bb88280, extra_info="") [2024-04-24 10:00:48.245979] INFO [LIB] stat (utility.h:1197) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F27D-0-0] [lt=161] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=31, stat_interval=5000000, avg (count/cost)=122710, this=0x14b14bb884b0, extra_info="") [2024-04-24 10:00:48.246026] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201362}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.246090] INFO [LIB] stat (utility.h:1197) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F27D-0-0] [lt=82] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=31, stat_interval=5000000, avg (count/cost)=0, this=0x14b14bb886e0, extra_info="") [2024-04-24 10:00:48.246204] INFO [LIB] stat (utility.h:1197) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F27D-0-0] [lt=86] [APPLY STAT CB EXECUTE TIME](cur_stat_count=31, stat_interval=5000000, avg (count/cost)=-713, this=0x14b14bb88910, extra_info="") [2024-04-24 10:00:48.246221] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=139][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924046605790000, v:0}, log_ts:{val:1713924046605790000, v:0}, lsn:{lsn:1002185585}, submit_ts:1713924046605790}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877456}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1640414, ctx_lock_wait_time=11) [2024-04-24 10:00:48.246295] INFO [LIB] stat (utility.h:1197) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F27D-0-0] [lt=69] [APPLY STAT CB TOTAL TIME](cur_stat_count=31, stat_interval=5000000, avg (count/cost)=122138, this=0x14b14bb88b40, extra_info="") [2024-04-24 10:00:48.246425] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD27D-0-0] [lt=101][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1001, self="10.0.0.3:2882", server="10.0.0.2:2882", update_func={old_end_lsn:{lsn:88113112}, new_end_lsn:{lsn:88115080}, old_advance_time_us:1713924046757322, new_ack_time_us:1713924048246418, advance delay(us):1489096}) [2024-04-24 10:00:48.246442] WDIAG [CLOG] statistics_cb_cost_ (ob_log_apply_service.cpp:933) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F27D-0-0] [lt=138][errcode=-4389] cb cost too much time(lsn={lsn:79584822}, scn={val:1713924046872416000, v:0}, idx=0, total_cost_time=1371189, append_cost_time=17, cb_wait_thread_time=1371999, cb_wait_commit_time=0, cb_cost_time=-827, append_start_time=1713924046873388, append_finish_time=1713924046873405, cb_first_handle_time=1713924048245404, cb_finish_time=1713924048244577) [2024-04-24 10:00:48.246590] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD27D-0-0] [lt=100][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=330719, log_task={header:{begin_lsn:{lsn:88113112}, end_lsn:{lsn:88115080}, log_id:330719, min_scn:{val:1713924047095221000, v:0}, max_scn:{val:1713924047226097000, v:0}, data_len:1912, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:88113112}, data_checksum:26217130, accum_checksum:167963847, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924047095518, freeze_ts:1713924047759180, submit_ts:1713924047759361, flushed_ts:1713924047768680, gen_to_freeze cost time:663662, gen_to_submit cost time:663843, submit_to_flush cost time:9319}, fs_cb_begin_ts=1713924048246533, log_life_time=1151015) [2024-04-24 10:00:48.246643] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201036}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.246722] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F27D-0-0] [lt=110][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924047095221000, v:0}, log_ts:{val:1713924047095221000, v:0}, lsn:{lsn:88113168}, submit_ts:1713924047095221}, this:0x14b148378f80, is_inited_:true, trans_id:{txid:327173}, ls_id:{id:1001}, ctx:0x14b148376010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1151478, ctx_lock_wait_time=10) [2024-04-24 10:00:48.246807] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2C208F1CE-0-0] [lt=83][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924046589452001, v:0}, log_ts:{val:1713924046589452001, v:0}, lsn:{lsn:79580890}, submit_ts:1713924046589452}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176904}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1657340, ctx_lock_wait_time=8) [2024-04-24 10:00:48.247075] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2C0D8F349-0-0] [lt=70][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924047180037000, v:0}, log_ts:{val:1713924047180037000, v:0}, lsn:{lsn:88113638}, submit_ts:1713924047180037}, this:0x14b148372200, is_inited_:true, trans_id:{txid:327177}, ls_id:{id:1001}, ctx:0x14b14836f290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1067019, ctx_lock_wait_time=11) [2024-04-24 10:00:48.247223] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200278}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.247306] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2C198F24F-0-0] [lt=117][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:0, v:0}, log_ts:{val:1713924046589452002, v:0}, lsn:{lsn:79581303}, submit_ts:1713924046707343}, this:0x14b148002428, is_inited_:true, trans_id:{txid:176906}, ls_id:{id:1}, ctx:0x14b147fffbd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:true, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1539950, ctx_lock_wait_time=6) [2024-04-24 10:00:48.247369] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2C0D8F34B-0-0] [lt=119][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924047193409000, v:0}, log_ts:{val:1713924047193409000, v:0}, lsn:{lsn:88114098}, submit_ts:1713924047193409}, this:0x14b14836b480, is_inited_:true, trans_id:{txid:327178}, ls_id:{id:1001}, ctx:0x14b148368510, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1053935, ctx_lock_wait_time=14) [2024-04-24 10:00:48.247588] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2C0D8F34D-0-0] [lt=81][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924047226097000, v:0}, log_ts:{val:1713924047226097000, v:0}, lsn:{lsn:88114604}, submit_ts:1713924047226097}, this:0x14b148367dc0, is_inited_:true, trans_id:{txid:327179}, ls_id:{id:1001}, ctx:0x14b148364e50, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1021472, ctx_lock_wait_time=6) [2024-04-24 10:00:48.247596] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=125][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924046881252000, v:0}, log_ts:{val:1713924046881252000, v:0}, lsn:{lsn:314127641}, submit_ts:1713924046881252}, this:0x14b15a130680, is_inited_:true, trans_id:{txid:568559}, ls_id:{id:1}, ctx:0x14b15a12d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1366323, ctx_lock_wait_time=13) [2024-04-24 10:00:48.247831] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200928}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.247960] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=191][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924046942380000, v:0}, log_ts:{val:1713924046942380000, v:0}, lsn:{lsn:314128351}, submit_ts:1713924046942380}, this:0x14b15a185f40, is_inited_:true, trans_id:{txid:568560}, ls_id:{id:1}, ctx:0x14b15a182fd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1305560, ctx_lock_wait_time=9) [2024-04-24 10:00:48.248203] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=91][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924046799990000, v:0}, log_ts:{val:1713924046799990000, v:0}, lsn:{lsn:314127056}, submit_ts:1713924046799990}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:391371}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1448200, ctx_lock_wait_time=6) [2024-04-24 10:00:48.248414] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=153] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200033}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.248424] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=69] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2024-04-24 10:00:48.249065] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200326}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.249540] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200114}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.250147] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=54] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200398}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.250441] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:159) [861548][OmtNodeBalancer][T1001][YB420A000003-000616C2A8291461-0-0] [lt=100] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2024-04-24 10:00:48.250591] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [861548][OmtNodeBalancer][T1001][YB420A000003-000616C2A8291461-0-0] [lt=96] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2024-04-24 10:00:48.250676] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:131) [861548][OmtNodeBalancer][T1001][YB420A000003-000616C2A8291461-0-0] [lt=74] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2024-04-24 10:00:48.250697] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201288}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.250741] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [862399][T1001_ReqMemEvi][T1001][YB420A000003-000616C2B05929A4-0-0] [lt=62] slow inner sql(last_ret=0, sql={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, process_time=1285429) [2024-04-24 10:00:48.250761] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1457) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B39C6512-0-0] [lt=110] receive register_req from acceptor, ignore(self="10.0.0.3:2882", palf_id=1, child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924048245354, keepalive_ts:-1}) [2024-04-24 10:00:48.250860] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1476) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B39C6512-0-0] [lt=84] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="10.0.0.3:2882", child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924048245354, keepalive_ts:-1}, is_to_leader=true) [2024-04-24 10:00:48.251021] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [862399][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=86] get number(ret=0, service_type_=0, range=1, base_id=1713924048250422000, start_id=1713924048250422000, end_id=1713924048250422001) [2024-04-24 10:00:48.251299] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [862399][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=106] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=2223) [2024-04-24 10:00:48.251391] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200894}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.251440] INFO [LIB] stat (utility.h:1197) [862399][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=92] [PALF STAT APPEND COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=0, this=0x14b14c97ee50, extra_info="id:1") [2024-04-24 10:00:48.252032] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200172}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.252543] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200656}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.253315] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201478}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.253591] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B47F14AD-0-0] [lt=161][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=301032, log_task={header:{begin_lsn:{lsn:165099068}, end_lsn:{lsn:165099190}, log_id:301032, min_scn:{val:1713924047274777000, v:0}, max_scn:{val:1713924047274777000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:165097495}, prev_proposal_id:5, committed_end_lsn:{lsn:165099068}, data_checksum:1715074492, accum_checksum:2841548286, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924047758427, freeze_ts:1713924047758440, submit_ts:1713924047758598, flushed_ts:1713924047773959, gen_to_freeze cost time:13, gen_to_submit cost time:171, submit_to_flush cost time:15361}, fs_cb_begin_ts=1713924048253543, log_life_time=495116) [2024-04-24 10:00:48.253842] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200466}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.254366] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200516}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.255051] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201390}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.255552] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201276}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.255849] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=76][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179740129}], scn_array=[{val:1713924047744490002, v:0}], curr_size=531, accum_size=3383, time_cost=11435) [2024-04-24 10:00:48.256392] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=123] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200028}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.256462] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:860) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BEEED977-0-0] [lt=55] memtable handle rollback to successfuly(from_seq_no=1713924048256177, to_seq_no=1713924048246905, *this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877458} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=1, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}) [2024-04-24 10:00:48.256924] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201132}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.257448] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201392}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.258031] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200380}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.258572] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201072}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.258678] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=70][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:40357212}], scn_array=[{val:1713924047781305001, v:0}], curr_size=706, accum_size=1155, time_cost=12446) [2024-04-24 10:00:48.259225] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201216}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.259474] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:122) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=71] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2024-04-24 10:00:48.259771] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200622}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.259956] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=67][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.260450] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200268}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.260442] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [874525][T1004_ReqMemEvi][T1004][Y0-0000000000000000-0-0] [lt=101] sql audit evict task end(request_manager_->get_tenant_id()=1004, evict_high_mem_level=278313880, evict_high_size_level=90000, evict_batch_count=0, elapse_time=7, size_used=80473, mem_used=131023872) [2024-04-24 10:00:48.260784] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [862260][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=117] no need to update min/max version(new_version={val:1713924045845177000, v:0}, min_version={val:1713924045845177000, v:0}, max_version={val:1713924045845177000, v:0}) [2024-04-24 10:00:48.260974] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201010}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.261405] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200958}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.261601] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:159) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=111] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2024-04-24 10:00:48.261915] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201488}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.262360] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200029}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.262741] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2A1-0-0] [lt=126] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, append size=4470) [2024-04-24 10:00:48.262854] INFO [LIB] stat (utility.h:1197) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2A1-0-0] [lt=91] [PALF STAT APPEND COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=0, this=0x14b0c0f0ae50, extra_info="id:1001") [2024-04-24 10:00:48.262832] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200334}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.263772] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=286] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200041}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.264446] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200514}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.264812] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:196) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=72] refresh_resource_function_mapping_rule(tenant_id=1002, plan=, rules=[]) [2024-04-24 10:00:48.264923] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:100) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=103] refresh resource mapping rule success(tenant_id=1002, plan=) [2024-04-24 10:00:48.265015] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200736}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.265107] INFO [STORAGE.TRANS] rollback_to_savepoint (ob_trans_part_ctx.cpp:7510) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BEEED977-0-0] [lt=84] rollback succeed trivially(op_sn=4, to_scn=1713924048246905, last_scn=1713924048246905) [2024-04-24 10:00:48.265507] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200680}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.266127] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200900}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.266951] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=128] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200682}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.267035] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:159) [861548][OmtNodeBalancer][T1002][YB420A000003-000616C2A8291461-0-0] [lt=58] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2024-04-24 10:00:48.267195] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [861548][OmtNodeBalancer][T1002][YB420A000003-000616C2A8291461-0-0] [lt=97] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2024-04-24 10:00:48.267295] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:131) [861548][OmtNodeBalancer][T1002][YB420A000003-000616C2A8291461-0-0] [lt=92] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2024-04-24 10:00:48.267489] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200816}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.268155] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201238}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.268736] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200336}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.269038] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [862437][T1001_LSArchive][T1001][YB420A000003-000616C2AEF8F0E7-0-0] [lt=74] gc stale ls task succ [2024-04-24 10:00:48.269480] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200240}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.270081] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=106] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201084}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.270361] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [2323545][T1004_L0_G0][T1004][YB420A000003-000616C2C1B8F18D-0-0] [lt=216] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924036937468, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924048269563, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:48.270649] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=120] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200750}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.271341] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200984}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.272010] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200214}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.272221] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:122) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=90] refresh resource plan success(tenant_id=1003, plan_name=, directives=[]) [2024-04-24 10:00:48.272562] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200222}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.273043] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=112] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.273162] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=95] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.273243] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201192}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.273253] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=81] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.274103] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201052}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.274671] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=126] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200328}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.274692] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:159) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=95] refresh resource user mapping rule(tenant_id=1003, plan=, user_rules=[]) [2024-04-24 10:00:48.275473] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=128] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200784}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.276117] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200162}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.276738] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201304}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.277194] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:196) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=128] refresh_resource_function_mapping_rule(tenant_id=1003, plan=, rules=[]) [2024-04-24 10:00:48.277356] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:100) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=151] refresh resource mapping rule success(tenant_id=1003, plan=) [2024-04-24 10:00:48.277476] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2405) [2325201][T1001_L0_G19][T1001][YB420A000002-000616C2B22C2732-0-0] [lt=64] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"10.0.0.2:2882", ls_array_:[{id:1}]}) [2024-04-24 10:00:48.277519] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=174] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200676}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.277610] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1509) [2325201][T1001_L0_G19][T1001][YB420A000002-000616C2B22C2732-0-0] [lt=89] get_member_gc_stat(is_valid_member=true, stat=1, member_list=2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="10.0.0.2:2882", this={role:1, proposal_id:5, palf_env_:0x14b157ff2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2024-04-24 10:00:48.278278] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201464}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.278988] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200728}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.279498] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200606}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.280200] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=125] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201274}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.280756] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201148}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.281330] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201142}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.281953] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200580}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.282548] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201214}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.283125] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=136][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:314130306}], scn_array=[{val:1713924048268280000, v:0}], curr_size=468, accum_size=3840, time_cost=13102) [2024-04-24 10:00:48.283371] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=137] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201038}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.284001] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200198}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.284543] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200166}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.285258] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=130] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200232}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.285825] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:1152921504606846987}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.286374] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=133] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200218}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.287012] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200842}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.287542] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201128}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.288110] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200724}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.288603] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200390}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.289201] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200212}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.289805] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201468}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.290415] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=179] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200824}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.291036] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200032}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.291490] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200352}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.291993] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201080}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.292421] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200624}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.292993] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200998}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.293431] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200616}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.294027] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200786}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.294523] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200230}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.295070] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200046}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.295536] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201314}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.296089] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200670}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.296559] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200916}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.297114] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200532}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.297577] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200820}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.298170] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201110}, multi_version_start=1713895481911822000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.298697] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200470}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.299381] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201022}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.299931] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200226}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.300375] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201404}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.300972] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200720}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.301487] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200586}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.302040] INFO [LIB] stat (utility.h:1197) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=141] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=149, this=0x14b1470ab470, extra_info="") [2024-04-24 10:00:48.302096] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200306}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.302147] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=78] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=2, avg_log_batch_cnt=1, total_group_log_size=244, avg_group_log_size=122) [2024-04-24 10:00:48.302247] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=234) [2024-04-24 10:00:48.302556] INFO [LIB] stat (utility.h:1197) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=104] [PALF STAT APPEND COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=0, this=0x14b147074e50, extra_info="id:1") [2024-04-24 10:00:48.302649] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=82] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.302727] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200134}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.302749] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=78] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.302844] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=87] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.303275] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200944}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.303759] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200858}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.304343] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200266}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.304991] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=126] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201058}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.305464] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200590}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.306027] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200738}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.306546] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=125] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201368}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.307130] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200954}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.307573] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200968}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.308093] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201286}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.308600] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=109] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200548}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.308967] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=92] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8892784640, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:48.309184] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200536}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.309612] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201268}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.310130] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200372}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.310644] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200458}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.311117] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201340}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.311692] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200402}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.312254] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200568}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.312727] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200774}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.313302] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200652}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.313945] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200260}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.314459] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200932}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.315077] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200884}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.315636] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201046}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.316224] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200180}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.316818] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200044}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.317314] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=136] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200754}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.317650] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=143] current clock is (clock_=0, max_seq_=0, mem_limit=3181127926, cur_mem_hold=297795584, attr_.tenant_id_=1004) [2024-04-24 10:00:48.317801] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200800}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.318354] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=53] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201176}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.318527] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:607) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=103] sql stat record not found(ret=-5138, key={sql_id:"3B96A400E13ADA9EC17191005B832B6", plan_hash:3290318591324336132, source_addr:"10.0.0.2:58274"}) [2024-04-24 10:00:48.318996] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200642}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.319577] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200080}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.319985] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=126] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1003, min_version={val:1713924045997782001, v:0}, base_version={val:1713924045997782001, v:0}, server_count=2, svr_array=[{addr:"10.0.0.2:2882", version:{val:1713924045997782001, v:0}, valid_part_count:0, total_part_count:1, generate_tstamp:1713924048310855, is_skipped:true}, {addr:"10.0.0.3:2882", version:{val:1713924045997782000, v:0}, valid_part_count:0, total_part_count:1, generate_tstamp:1713924048319701, is_skipped:true}], skip_server_count=2, skip_servers=["10.0.0.2:2882", "10.0.0.3:2882"]) [2024-04-24 10:00:48.320143] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=148] no need to update min/max version(new_version={val:1713924045997782001, v:0}, min_version={val:1713924045997782001, v:0}, max_version={val:1713924045997782001, v:0}) [2024-04-24 10:00:48.320320] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200716}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.320963] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201296}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.321480] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201158}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.322090] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200688}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.322607] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200952}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.322643] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [874526][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=119] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1004, min_version={val:1713924045938980001, v:0}, base_version={val:1713924045938980001, v:0}, server_count=2, svr_array=[{addr:"10.0.0.2:2882", version:{val:1713924046087776000, v:0}, valid_part_count:0, total_part_count:3, generate_tstamp:1713924048270466, is_skipped:true}, {addr:"10.0.0.3:2882", version:{val:1713924046083050000, v:0}, valid_part_count:0, total_part_count:3, generate_tstamp:1713924048322533, is_skipped:true}], skip_server_count=2, skip_servers=["10.0.0.2:2882", "10.0.0.3:2882"]) [2024-04-24 10:00:48.322737] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [874526][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=88] no need to update min/max version(new_version={val:1713924045938980001, v:0}, min_version={val:1713924045938980001, v:0}, max_version={val:1713924045938980001, v:0}) [2024-04-24 10:00:48.323234] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200524}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.323740] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200370}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.324218] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200660}, multi_version_start=1713895482024969001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.324702] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200446}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.325297] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200770}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.325799] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200980}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.326435] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201200}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.326834] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.326995] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201008}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.327454] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201070}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.327962] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200384}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.327983] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=76] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.328546] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201026}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.329065] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200780}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.329698] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200558}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.330353] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200804}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.330975] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=124] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201486}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.331718] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=289] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200522}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.332305] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200870}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.332936] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=104] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200752}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.333367] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200818}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.333949] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201336}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.334423] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201188}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.334994] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201182}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.335451] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200158}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.336024] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [874739][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=85] ====== check clog disk timer task ====== [2024-04-24 10:00:48.336112] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200022}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.336130] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [874739][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=95] get_disk_usage(ret=0, capacity(MB):=3686, used(MB):=348) [2024-04-24 10:00:48.336288] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [874739][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=78] cannot_recycle_log_size statistics(cannot_recycle_log_size=231410180, threshold=1159641169, need_update_checkpoint_scn=false) [2024-04-24 10:00:48.336721] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200224}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.337317] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201024}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.338013] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200152}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.338506] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200650}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.339463] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=86] no need to update min/max version(new_version={val:1713924045694942002, v:0}, min_version={val:1713924045694942002, v:0}, max_version={val:1713924045694942002, v:0}) [2024-04-24 10:00:48.339657] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=699] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201124}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.340285] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201028}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.340796] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201356}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.341373] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=151] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200252}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.341965] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200726}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.342447] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:1152921504606846996}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.343018] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201042}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.343491] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200404}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.344096] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200722}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.344435] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=112] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.344606] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:1152921504606846991}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.345231] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200002}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.345736] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200004}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.346326] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200047}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.346957] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201120}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.347194] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=110][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:88119149}], scn_array=[{val:1713924048297356000, v:0}], curr_size=447, accum_size=6484, time_cost=48549) [2024-04-24 10:00:48.347666] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200912}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.348186] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200374}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.348625] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200826}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.348656] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [874735][T1003_ReqMemEvi][T1003][Y0-0000000000000000-0-0] [lt=58] sql audit evict task end(request_manager_->get_tenant_id()=1003, evict_high_mem_level=19327352, evict_high_size_level=90000, evict_batch_count=0, elapse_time=8, size_used=8058, mem_used=14558208) [2024-04-24 10:00:48.349145] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200840}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.349524] INFO [PALF] inner_write_once_ (log_block_handler.cpp:411) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=73] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=58179, this={dio_aligned_buf:{buf_write_offset:957, buf_padding_size:3139, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:3203371, truncate_used_ts:3918428}, log_block_size:67108864, dir_fd:159, io_fd:161}, aligned_buf_len=4096, aligned_buf="", aligned_block_offset=57344, buf_len=122, total_write_size_=6590, total_write_size_after_dio_=155648, ob_pwrite_used_ts=3321495, count=36, each_pwrite_cost=92263) [2024-04-24 10:00:48.349619] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200802}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.349665] INFO [LIB] stat (utility.h:1197) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=109] [PALF STAT WRITE LOG COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=229054, this=0x14b1470b53b0, extra_info="palf_id:1") [2024-04-24 10:00:48.349750] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=78][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67158851}], scn_array=[{val:1713924048276279000, v:0}], curr_size=122, accum_size=422, time_cost=47413) [2024-04-24 10:00:48.349813] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=57] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=422) [2024-04-24 10:00:48.349955] INFO [LIB] stat (utility.h:1214) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=106] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=278, this=0x14b1679f6530, extra_info="") [2024-04-24 10:00:48.350131] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200960}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.350124] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=95] [PALF STAT FLUSH CB COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=237, this=0x14b1470b55e0, extra_info="palf_id:1") [2024-04-24 10:00:48.350589] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200844}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.351076] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201418}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.351602] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201264}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.352336] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200904}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.352856] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200496}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.353485] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=132] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200280}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.354075] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200236}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.354272] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=91] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: MICRO_BLOCK_CACHE_IO, mode: read, size: 4096.00, iops: 11.79, rt: 700.67) [2024-04-24 10:00:48.354370] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=77] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:48.354399] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F6F-0-0] [lt=119][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=482544, log_task={header:{begin_lsn:{lsn:298514948}, end_lsn:{lsn:298515446}, log_id:482544, min_scn:{val:1713924047736671000, v:0}, max_scn:{val:1713924047736671000, v:0}, data_len:442, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:298513336}, data_checksum:2553231233, accum_checksum:3966178707, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924047737031, freeze_ts:1713924047745232, submit_ts:1713924047745356, flushed_ts:1713924047756131, gen_to_freeze cost time:8201, gen_to_submit cost time:8325, submit_to_flush cost time:10775}, fs_cb_begin_ts=1713924048354324, log_life_time=617293) [2024-04-24 10:00:48.354591] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:1152921504606846982}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.354662] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=112][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924047754365000, v:0}, log_ts:{val:1713924047754365000, v:0}, lsn:{lsn:298515502}, submit_ts:1713924047754365}, this:0x14b0fc0a5ac0, is_inited_:true, trans_id:{txid:548719}, ls_id:{id:1}, ctx:0x14b0fc0a2b50, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=600280, ctx_lock_wait_time=8) [2024-04-24 10:00:48.355047] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=158] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.98, rt: 1094.00) [2024-04-24 10:00:48.355056] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=113][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924047736671000, v:0}, log_ts:{val:1713924047736671000, v:0}, lsn:{lsn:298515004}, submit_ts:1713924047736671}, this:0x14b0fbe0a680, is_inited_:true, trans_id:{txid:548718}, ls_id:{id:1}, ctx:0x14b0fbe07710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=618361, ctx_lock_wait_time=7) [2024-04-24 10:00:48.355131] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200846}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.355173] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=100] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:48.355667] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200276}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.355920] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:159) [861548][OmtNodeBalancer][T1003][YB420A000003-000616C2A8291461-0-0] [lt=90] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2024-04-24 10:00:48.356016] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [861548][OmtNodeBalancer][T1003][YB420A000003-000616C2A8291461-0-0] [lt=60] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2024-04-24 10:00:48.356121] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:131) [861548][OmtNodeBalancer][T1003][YB420A000003-000616C2A8291461-0-0] [lt=97] tenant dfc status(ret=0, get_tenant_id()=1003, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2024-04-24 10:00:48.356212] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201328}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.356325] INFO [COMMON] print_io_status (ob_io_manager.cpp:1393) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=114] [IO STATUS SYS](tenant_id=1002, io_status=sys_group_name: SLOG_IO, mode: write, size: 8192.00, iops: 0.98, rt: 615183.00) [2024-04-24 10:00:48.356400] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:607) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=89] sql stat record not found(ret=-5138, key={sql_id:"42073B2EE6E34B3D43320FD6CDB8537", plan_hash:3290318591324336132, source_addr:"10.0.0.2:58374"}) [2024-04-24 10:00:48.356509] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=97] [IO STATUS CONFIG](tenant_id=1002, ref_cnt=2, io_config={group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:1}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:48.356937] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201358}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.357559] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=129] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201090}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.358334] INFO [COMMON] print_io_status (ob_io_manager.cpp:1393) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=184] [IO STATUS SYS](tenant_id=1004, io_status=sys_group_name: SLOG_IO, mode: write, size: 4096.00, iops: 0.98, rt: 697096.00) [2024-04-24 10:00:48.358479] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=124] [IO STATUS CONFIG](tenant_id=1004, ref_cnt=3, io_config={group_num_:0, memory_limit_:11596411700, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:2}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:48.358616] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=110] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-24 10:00:48.358708] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=472] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201330}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.358770] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=139] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-24 10:00:48.358845] INFO [CLOG] run1 (ob_garbage_collector.cpp:1338) [862415][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=135] Garbage Collector is running(seq_=4860, gc_interval=10000000) [2024-04-24 10:00:48.359016] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1473) [862415][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=100] self is leader, skip it(ls->get_ls_id()={id:1}) [2024-04-24 10:00:48.359171] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1431) [862415][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=123] gc_check_member_list_ cost time(ret=0, time_us=210) [2024-04-24 10:00:48.359308] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=137] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201074}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.359300] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [862415][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=80] execute_gc cost time(ret=0, time_us=8) [2024-04-24 10:00:48.360101] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [874495][T1004_OB_SLOG][T1004][Y0-0000000000000000-0-0] [lt=81] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=93233, offset=21596421}, end_cursor:ObLogCursor{file_id=1, log_id=93234, offset=21596662}, is_inited:true, is_local:false, buf_size:8192, buf:0x14b19ef10050, len:1787, log_data_len:241, seq:93233, flush_finish:false, flush_ret:0}) [2024-04-24 10:00:48.360143] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200518}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.360744] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201164}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.360946] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:122) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=84] refresh resource plan success(tenant_id=1004, plan_name=, directives=[]) [2024-04-24 10:00:48.361550] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=141][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.361635] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=202] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200188}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.362326] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200940}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.362544] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:954) [861320][SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=103] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=669876224, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=403865600, wash_ratio=6, tenant_id=1001) [2024-04-24 10:00:48.362788] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:323) [861320][SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=137] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2024-04-24 10:00:48.362959] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1202) [861320][SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=163] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=162, drift_size_=0, pre_drift_size=0, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2024-04-24 10:00:48.362973] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201204}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.363012] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [862260][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=75] get number(ret=0, service_type_=0, range=1, base_id=1713924048362755000, start_id=1713924048362755000, end_id=1713924048362755001) [2024-04-24 10:00:48.363075] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:954) [861320][SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=73] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=963121152, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=110620672, wash_ratio=6, tenant_id=1002) [2024-04-24 10:00:48.363288] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:323) [861320][SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=109] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2024-04-24 10:00:48.363329] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD1418-0-0] [lt=85][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=364236, log_task={header:{begin_lsn:{lsn:179740129}, end_lsn:{lsn:179740660}, log_id:364236, min_scn:{val:1713924047744490002, v:0}, max_scn:{val:1713924047771637000, v:0}, data_len:475, proposal_id:5, prev_lsn:{lsn:179739101}, prev_proposal_id:5, committed_end_lsn:{lsn:179740129}, data_checksum:809875915, accum_checksum:2900025543, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924048244112, freeze_ts:1713924048244122, submit_ts:1713924048244191, flushed_ts:1713924048258350, gen_to_freeze cost time:10, gen_to_submit cost time:79, submit_to_flush cost time:14159}, fs_cb_begin_ts=1713924048363203, log_life_time=119091) [2024-04-24 10:00:48.363398] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1202) [861320][SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=98] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=926, drift_size_=0, pre_drift_size=0, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2024-04-24 10:00:48.363525] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:954) [861320][SqlMemTimer][T1003][Y0-0000000000000000-0-0] [lt=89] trace max work area(auto_calc=true, tenant_max_memory_limit=1288490188, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=64424509, max_wa_memory_size=64424509, tmp_max_wa_memory_size=64424509, pre_mem_target=64424509, remain_memory_size=64424509, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=671387648, washable_size=-2, max_workarea_memory_size=64424509, max_tenant_memory_size=617102540, wash_ratio=6, tenant_id=1003) [2024-04-24 10:00:48.363543] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201174}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.363582] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C3B-0-0] [lt=107][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=673415, log_task={header:{begin_lsn:{lsn:1002187102}, end_lsn:{lsn:1002187214}, log_id:673415, min_scn:{val:1713924046605790001, v:0}, max_scn:{val:1713924046605790001, v:0}, data_len:56, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1002185529}, data_checksum:3578201321, accum_checksum:3810436167, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924047745667, freeze_ts:1713924047750277, submit_ts:1713924047750402, flushed_ts:1713924047757694, gen_to_freeze cost time:4610, gen_to_submit cost time:4735, submit_to_flush cost time:7292}, fs_cb_begin_ts=1713924048363525, log_life_time=617858) [2024-04-24 10:00:48.363689] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:323) [861320][SqlMemTimer][T1003][Y0-0000000000000000-0-0] [lt=80] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=8053063, calc_global_bound_size=1048576000000, mem_target_=64424509, wa_max_memory_size=64424509, profile_cnt_=0, total_memory_size=0, max_wa_size=64424509, avg_bound_size=8053063, max_bound_size=8053063, min_bound_size_=1048576) [2024-04-24 10:00:48.363976] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD27E-0-0] [lt=109][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=299913, log_task={header:{begin_lsn:{lsn:40357212}, end_lsn:{lsn:40357918}, log_id:299913, min_scn:{val:1713924047781305001, v:0}, max_scn:{val:1713924048022376000, v:0}, data_len:650, proposal_id:5, prev_lsn:{lsn:40356763}, prev_proposal_id:5, committed_end_lsn:{lsn:40357212}, data_checksum:1954685642, accum_checksum:1599028303, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924048244831, freeze_ts:1713924048244837, submit_ts:1713924048244934, flushed_ts:1713924048258977, gen_to_freeze cost time:6, gen_to_submit cost time:103, submit_to_flush cost time:14043}, fs_cb_begin_ts=1713924048363742, log_life_time=118911) [2024-04-24 10:00:48.364194] INFO [CLOG] run1 (ob_garbage_collector.cpp:1338) [874720][T1003_GCCollect][T1003][Y0-0000000000000000-0-0] [lt=121] Garbage Collector is running(seq_=4667, gc_interval=10000000) [2024-04-24 10:00:48.364376] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:188) [862224][T1002_FreInfoRe][T1002][Y0-0000000000000000-0-0] [lt=143] submit reserved snapshot log success(ls_id={id:1001}, new_snapshot_version=1713922241263293000) [2024-04-24 10:00:48.364407] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200814}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.364428] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1473) [874720][T1003_GCCollect][T1003][Y0-0000000000000000-0-0] [lt=106] self is leader, skip it(ls->get_ls_id()={id:1}) [2024-04-24 10:00:48.364446] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:159) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=112] refresh resource user mapping rule(tenant_id=1004, plan=, user_rules=[]) [2024-04-24 10:00:48.364488] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1071) [862224][T1002_FreInfoRe][T1002][Y0-0000000000000000-0-0] [lt=70] update reserved snapshot finished(cost_ts=1515804, reserved_snapshot=1713922241263293000) [2024-04-24 10:00:48.364598] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1431) [874720][T1003_GCCollect][T1003][Y0-0000000000000000-0-0] [lt=114] gc_check_member_list_ cost time(ret=0, time_us=280) [2024-04-24 10:00:48.364738] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [861714][T1_ReplaySrv6][T1][Y0-0000000000000000-0-0] [lt=138] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924036825066, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924048362755, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:48.364754] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [874720][T1003_GCCollect][T1003][Y0-0000000000000000-0-0] [lt=125] execute_gc cost time(ret=0, time_us=9) [2024-04-24 10:00:48.365105] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=104] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200556}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.365703] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=118] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200636}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.366394] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=106] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200298}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.363806] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1202) [861320][SqlMemTimer][T1003][Y0-0000000000000000-0-0] [lt=103] timer to calc global bound size(ret=0, global_bound_size_=8053063, manual_calc_cnt_=109, drift_size_=0, pre_drift_size=0, wa_max_memory_size=64424509, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1003, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=8053063, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=64424509, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2024-04-24 10:00:48.366977] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:954) [861320][SqlMemTimer][T1004][Y0-0000000000000000-0-0] [lt=3112] trace max work area(auto_calc=true, tenant_max_memory_limit=11596411700, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=579820585, max_wa_memory_size=579820585, tmp_max_wa_memory_size=579820585, pre_mem_target=577723433, remain_memory_size=579820585, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1648771072, washable_size=-2, max_workarea_memory_size=579820585, max_tenant_memory_size=9947640628, wash_ratio=6, tenant_id=1004) [2024-04-24 10:00:48.367005] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:196) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=160] refresh_resource_function_mapping_rule(tenant_id=1004, plan=, rules=[]) [2024-04-24 10:00:48.367042] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201446}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.367088] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:100) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=75] refresh resource mapping rule success(tenant_id=1004, plan=) [2024-04-24 10:00:48.367135] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:323) [861320][SqlMemTimer][T1004][Y0-0000000000000000-0-0] [lt=81] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=72477573, calc_global_bound_size=1048576000000, mem_target_=579820585, wa_max_memory_size=579820585, profile_cnt_=0, total_memory_size=0, max_wa_size=579820585, avg_bound_size=72477573, max_bound_size=72477573, min_bound_size_=1048576) [2024-04-24 10:00:48.367233] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1202) [861320][SqlMemTimer][T1004][Y0-0000000000000000-0-0] [lt=90] timer to calc global bound size(ret=0, global_bound_size_=72477573, manual_calc_cnt_=640, drift_size_=0, pre_drift_size=14362880, wa_max_memory_size=579820585, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1004, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=72477573, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=579820585, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2024-04-24 10:00:48.367603] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=85][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924047777372000, v:0}, log_ts:{val:1713924047777372000, v:0}, lsn:{lsn:1002187270}, submit_ts:1713924047777372}, this:0x14b168f33d40, is_inited_:true, trans_id:{txid:877457}, ls_id:{id:1001}, ctx:0x14b168f30dd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=590215, ctx_lock_wait_time=7) [2024-04-24 10:00:48.367603] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200782}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.368149] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:834) [862160][T1002_BalanceEx][T1002][YB420A000003-000616C2B3D9AD9F-0-0] [lt=131] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2024-04-24 10:00:48.368327] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:171) [862160][T1002_BalanceEx][T1002][YB420A000003-000616C2B3D9AD9F-0-0] [lt=167] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2024-04-24 10:00:48.368383] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=173] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200970}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.368946] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200202}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.369467] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201122}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.370056] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201382}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.370077] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:159) [861548][OmtNodeBalancer][T1004][YB420A000003-000616C2A8291461-0-0] [lt=92] trace tenant dfc parameters(max_parallel_cnt_=80, max_blocked_buffer_size_=26214400, max_buffer_size_=52428800) [2024-04-24 10:00:48.370216] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [861548][OmtNodeBalancer][T1004][YB420A000003-000616C2A8291461-0-0] [lt=98] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2024-04-24 10:00:48.370302] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:131) [861548][OmtNodeBalancer][T1004][YB420A000003-000616C2A8291461-0-0] [lt=79] tenant dfc status(ret=0, get_tenant_id()=1004, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=80, get_max_blocked_buffer_size()=26214400, get_max_buffer_size()=52428800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2024-04-24 10:00:48.370483] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:109) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=121] refresh tenant units(sys_unit_cnt=2, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3}}, mode:0, create_timestamp:1713875175740107, is_removed:false}, {tenant_id:1002, unit_id:1002, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"ocp_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"5.4GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:1}}, mode:0, create_timestamp:1713875391883804, is_removed:false}, {tenant_id:1001, unit_id:1002, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"ocp_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.6GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:1}}, mode:0, create_timestamp:1713875391883804, is_removed:false}, {tenant_id:1004, unit_id:1003, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1004, name:"c8", resource:{min_cpu:2, max_cpu:8, memory_size:"10.8GB", log_disk_size:"32.4GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:2}}, mode:0, create_timestamp:1713877325237012, is_removed:false}, {tenant_id:1003, unit_id:1003, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1004, name:"c8", resource:{min_cpu:1, max_cpu:1, memory_size:"1.2GB", log_disk_size:"3.6GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:2}}, mode:0, create_timestamp:1713877325237012, is_removed:false}], ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:48.370638] INFO set_nway (ob_tenant_mutil_allocator.cpp:335) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=125] finish set nway(tenant_id_=1, nway=3) [2024-04-24 10:00:48.370692] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD27F-0-0] [lt=139][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=330720, log_task={header:{begin_lsn:{lsn:88115080}, end_lsn:{lsn:88117460}, log_id:330720, min_scn:{val:1713924047765133000, v:0}, max_scn:{val:1713924047765133000, v:0}, data_len:2324, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:88113112}, data_checksum:1157009444, accum_checksum:1332675691, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924047766510, freeze_ts:1713924047768706, submit_ts:1713924047768835, flushed_ts:1713924047774976, gen_to_freeze cost time:2196, gen_to_submit cost time:2325, submit_to_flush cost time:6141}, fs_cb_begin_ts=1713924048370608, log_life_time=604098) [2024-04-24 10:00:48.370731] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=107] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201334}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.370753] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:380) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=85] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=536870900, pre_tma_limit=536870900, cur_memstore_limit_percent=50, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3}}, mode:0, create_timestamp:1713875175740107, is_removed:false}) [2024-04-24 10:00:48.370905] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:391) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=99] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2024-04-24 10:00:48.370989] INFO set_nway (ob_tenant_mutil_allocator.cpp:335) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=67] finish set nway(tenant_id_=1002, nway=1) [2024-04-24 10:00:48.371080] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:380) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=84] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=1, new_tma_limit=536870900, pre_tma_limit=536870900, cur_memstore_limit_percent=50, tenant_config={tenant_id:1002, unit_id:1002, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"ocp_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"5.4GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:1}}, mode:0, create_timestamp:1713875391883804, is_removed:false}) [2024-04-24 10:00:48.371008] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1626) [874720][T1003_GCCollect][T1003][Y0-0000000000000000-0-0] [lt=106] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2024-04-24 10:00:48.371176] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:391) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=89] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2024-04-24 10:00:48.371210] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1648) [874720][T1003_GCCollect][T1003][Y0-0000000000000000-0-0] [lt=168] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2024-04-24 10:00:48.371242] INFO set_nway (ob_tenant_mutil_allocator.cpp:335) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=53] finish set nway(tenant_id_=1001, nway=1) [2024-04-24 10:00:48.371235] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2C0D8F350-0-0] [lt=185][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924047765133000, v:0}, log_ts:{val:1713924047765133000, v:0}, lsn:{lsn:88115136}, submit_ts:1713924047765133}, this:0x14b1481bf8c0, is_inited_:true, trans_id:{txid:327181}, ls_id:{id:1001}, ctx:0x14b1481bc950, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=606090, ctx_lock_wait_time=6) [2024-04-24 10:00:48.371311] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [874720][T1003_GCCollect][T1003][Y0-0000000000000000-0-0] [lt=69] execute_gc cost time(ret=0, time_us=114) [2024-04-24 10:00:48.371318] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200322}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.371325] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:380) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=76] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=536870900, pre_tma_limit=536870900, cur_memstore_limit_percent=50, tenant_config={tenant_id:1001, unit_id:1002, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"ocp_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.6GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:1}}, mode:0, create_timestamp:1713875391883804, is_removed:false}) [2024-04-24 10:00:48.371393] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [874720][T1003_GCCollect][T1003][Y0-0000000000000000-0-0] [lt=63] ObSafeDestroyHandler start process [2024-04-24 10:00:48.371411] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:391) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=80] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2024-04-24 10:00:48.371625] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [874720][T1003_GCCollect][T1003][Y0-0000000000000000-0-0] [lt=140] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2024-04-24 10:00:48.371696] INFO set_nway (ob_tenant_mutil_allocator.cpp:335) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=266] finish set nway(tenant_id_=1004, nway=8) [2024-04-24 10:00:48.371811] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:380) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=108] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1004, nway=8, new_tma_limit=5798205850, pre_tma_limit=5798205850, cur_memstore_limit_percent=50, tenant_config={tenant_id:1004, unit_id:1003, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1004, name:"c8", resource:{min_cpu:2, max_cpu:8, memory_size:"10.8GB", log_disk_size:"32.4GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:2}}, mode:0, create_timestamp:1713877325237012, is_removed:false}) [2024-04-24 10:00:48.371855] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [874720][T1003_GCCollect][T1003][Y0-0000000000000000-0-0] [lt=145] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2024-04-24 10:00:48.371950] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1626) [862415][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=148] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2024-04-24 10:00:48.371998] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:391) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=179] succ to set_memstore_threshold of memstore allocator(tenant_id=1004, ret=0) [2024-04-24 10:00:48.372053] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1648) [862415][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=77] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2024-04-24 10:00:48.372083] INFO set_nway (ob_tenant_mutil_allocator.cpp:335) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=66] finish set nway(tenant_id_=1003, nway=1) [2024-04-24 10:00:48.372111] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201370}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.372147] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [862415][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=66] execute_gc cost time(ret=0, time_us=105) [2024-04-24 10:00:48.372153] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:380) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=64] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1003, nway=1, new_tma_limit=644245050, pre_tma_limit=644245050, cur_memstore_limit_percent=50, tenant_config={tenant_id:1003, unit_id:1003, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1004, name:"c8", resource:{min_cpu:1, max_cpu:1, memory_size:"1.2GB", log_disk_size:"3.6GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:2}}, mode:0, create_timestamp:1713877325237012, is_removed:false}) [2024-04-24 10:00:48.372225] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:391) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=62] succ to set_memstore_threshold of memstore allocator(tenant_id=1003, ret=0) [2024-04-24 10:00:48.372215] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [862415][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=55] ObSafeDestroyHandler start process [2024-04-24 10:00:48.372624] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200544}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.372942] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=74] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.373056] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=107] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.373230] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200768}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.373941] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=211][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179740660}], scn_array=[{val:1713924048242048000, v:0}], curr_size=1073, accum_size=4456, time_cost=10280) [2024-04-24 10:00:48.374160] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=111] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:1152921504606846992}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.374493] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:426) [862224][T1002_FreInfoRe][T1002][Y0-0000000000000000-0-0] [lt=159] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2024-04-24 10:00:48.374730] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200170}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.375368] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=219] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200560}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.375858] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=102][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:40357918}], scn_array=[{val:1713924048269563000, v:0}], curr_size=449, accum_size=1604, time_cost=11456) [2024-04-24 10:00:48.376035] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200512}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.376548] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200834}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.376575] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2C20C019D-0-0] [lt=136][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=301033, log_task={header:{begin_lsn:{lsn:165099190}, end_lsn:{lsn:165099312}, log_id:301033, min_scn:{val:1713924048246903000, v:0}, max_scn:{val:1713924048246903000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:165099068}, prev_proposal_id:5, committed_end_lsn:{lsn:165099190}, data_checksum:1715074492, accum_checksum:4140809815, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924048253431, freeze_ts:1713924048253443, submit_ts:1713924048253503, flushed_ts:1713924048259274, gen_to_freeze cost time:12, gen_to_submit cost time:72, submit_to_flush cost time:5771}, fs_cb_begin_ts=1713924048376516, log_life_time=123085) [2024-04-24 10:00:48.377275] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=109] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200570}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.377774] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201408}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.378344] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=171] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200710}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.378428] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:325) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=66] cal_all_part_disk_default_percentage succ(data_dir="/data/admin/test/oceanbase/store/sstable", clog_dir="/data/admin/test/oceanbase/store/clog", shared_mode=true, data_disk_total_size=1098974756864, data_disk_default_percentage=60, clog_disk_total_size=1098974756864, clog_disk_default_percentage=30) [2024-04-24 10:00:48.378507] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:361) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=73] decide disk size finished(suggested_disk_size=51539607552, suggested_disk_percentage=0, default_disk_percentage=30, total_space=1098974756864, disk_size=51539607552) [2024-04-24 10:00:48.378607] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=93] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=51539607552, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=51539607552, log_disk_percentage=0, total_log_disk_size=1098974756864) [2024-04-24 10:00:48.378801] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=179] try_resize success(ret=0, log_disk_size=51539607552, total_log_disk_size=1098974756864, this={dir::"/data/admin/test/oceanbase/store/clog/log_pool", dir_fd:14, meta_fd:15, log_pool_meta:{curr_total_size:51539607552, next_total_size:51539607552, status:0}, min_block_id:47, max_block_id:768, min_log_disk_size_for_all_tenants_:47244640256, is_inited:true}) [2024-04-24 10:00:48.378904] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201228}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.378953] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:133) [861548][OmtNodeBalancer][T0][YB420A000003-000616C2A8291461-0-0] [lt=146] refresh tenant config(tenants=[1, 1001, 1002, 1003, 1004], ret=0) [2024-04-24 10:00:48.379405] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201206}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.379957] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201472}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.380453] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200190}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.381179] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201226}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.381602] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201032}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.382160] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200498}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.382592] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200358}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.383262] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201440}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.383805] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200126}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.384421] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=196] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200766}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.385119] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=118] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200116}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.385655] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200042}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.386180] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200902}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.386617] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200574}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.387188] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200049}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.387658] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200140}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.388122] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200386}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.388521] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200672}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.389070] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200886}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.389567] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200610}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.390281] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200966}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.390469] INFO [PALF] check_parent_health (log_config_mgr.cpp:2744) [874414][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=70] re_register_parent reason(palf_id=1002, self="10.0.0.3:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1713924044379417, last_first_register_time_us=1713877519298378, register_time_us=1713924044379417, curr_time_us=1713924048390442) [2024-04-24 10:00:48.390639] INFO [PALF] register_parent_ (log_config_mgr.cpp:2601) [874414][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=110] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="10.0.0.3:2882", reason=1, leader="10.0.0.2:2882", register_time_us=1713924048390587, last_submit_register_req_time_us=1713924048390587, register_parent_reason=1) [2024-04-24 10:00:48.390746] INFO [PALF] check_parent_health (log_config_mgr.cpp:2753) [874414][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=99] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="10.0.0.3:2882") [2024-04-24 10:00:48.390844] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=135] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201076}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.391496] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=151] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200324}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.392055] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201462}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.392732] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=115] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200908}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.393444] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=122] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200396}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.394125] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200828}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.394962] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201310}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.395495] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200156}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.396224] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200027}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.396844] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200620}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.397579] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=183] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200474}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.398222] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=123] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200148}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.398806] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200242}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.399379] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=164] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201210}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.399987] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200866}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.400493] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200796}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.401075] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201490}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.401590] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200990}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.402157] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201066}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.402240] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=123] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.402371] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=106] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.402489] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=109] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.402917] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201212}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.403428] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201422}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.404087] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201096}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.404598] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201232}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.405193] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201260}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.405715] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200344}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.406258] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200026}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.406758] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200630}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.406817] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [861593][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=75] ====== tenant freeze timer task ====== [2024-04-24 10:00:48.407417] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201348}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.407957] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200790}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.408019] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:607) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=128] sql stat record not found(ret=-5138, key={sql_id:"B7A8B603CFAA9ACB88DB027E6CE3CA5", plan_hash:3290318591324336132, source_addr:"10.0.0.2:58280"}) [2024-04-24 10:00:48.408505] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200510}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.408986] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:201144}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.409468] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=53] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200924}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.410005] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200698}, multi_version_start=1713895482997014001, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.410479] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200144}, multi_version_start=1713895483108600000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.411043] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200023}, multi_version_start=1713895483108600000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.411570] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=107] get multi version start(ret=0, ls_id={id:1002}, tablet_id={id:200366}, multi_version_start=1713920415482750000, min_reserved_snapshot=1713922241263293000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241263293000, last_major_snapshot_version=1713895203744844000) [2024-04-24 10:00:48.411731] INFO [STORAGE] runTimerTask (ob_tenant_tablet_scheduler.cpp:174) [862274][T1002_SSTableGC][T1002][YB420A000003-000616C2B1F8F6E4-0-0] [lt=115] SSTableGCTask(cost_ts=1807034) [2024-04-24 10:00:48.414337] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=82] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1713924048414249], gts:1713924048413711000, latest_srr:[mts=1713924048414249]}) [2024-04-24 10:00:48.414404] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2325174][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C052F-0-0] [lt=53][errcode=-4023] tx is 2pc logging(this={this:0x14b14895b0d0, ref:4, trans_id:{txid:176909}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924078269409, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048273104], ctx_create_time:1713924048269563}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048269568, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048272353000, v:0}, log_ts:{val:1713924048272353000, v:0}, lsn:{lsn:79588596}, submit_ts:1713924048272353}, this:0x14b14895e040, is_inited_:true, trans_id:{txid:176909}, ls_id:{id:1}, ctx:0x14b14895b0d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176909}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048272353000, v:0}, start_scn:{val:1713924048272353000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465665976 max_table_version=1713877465684248 trans_version={val:1713924048272353000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176909} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1612204486 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048272353000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048272353, block_frozen_memtable:null}) [2024-04-24 10:00:48.414642] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2325174][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C052F-0-0] [lt=221][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b14895b0d0, ref:4, trans_id:{txid:176909}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924078269409, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048273104], ctx_create_time:1713924048269563}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048269568, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048272353000, v:0}, log_ts:{val:1713924048272353000, v:0}, lsn:{lsn:79588596}, submit_ts:1713924048272353}, this:0x14b14895e040, is_inited_:true, trans_id:{txid:176909}, ls_id:{id:1}, ctx:0x14b14895b0d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176909}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048272353000, v:0}, start_scn:{val:1713924048272353000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465665976 max_table_version=1713877465684248 trans_version={val:1713924048272353000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176909} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1612204486 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048272353000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048272353, block_frozen_memtable:null}) [2024-04-24 10:00:48.414797] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2325174][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C052F-0-0] [lt=145][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176909}) [2024-04-24 10:00:48.414853] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2325174][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C052F-0-0] [lt=49][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176909}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924048409930, cluster_id:1713874979}, expire_ts:1713924078188694, commit_start_scn:{val:1713924048269563000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:48.414973] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2325174][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C052F-0-0] [lt=103][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176909}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924048409930, cluster_id:1713874979}, expire_ts:1713924078188694, commit_start_scn:{val:1713924048269563000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:48.416244] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [861702][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=138] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:48.427018] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=133][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.428111] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=108] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.432193] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=130] current clock is (clock_=0, max_seq_=0, mem_limit=3181127926, cur_mem_hold=297795584, attr_.tenant_id_=1004) [2024-04-24 10:00:48.434916] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB8F9-0-0] [lt=77][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=505456, log_task={header:{begin_lsn:{lsn:314128793}, end_lsn:{lsn:314129808}, log_id:505456, min_scn:{val:1713924047767921000, v:0}, max_scn:{val:1713924047767921000, v:0}, data_len:959, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:314126934}, data_checksum:3530810297, accum_checksum:3526315183, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924047770071, freeze_ts:1713924047770088, submit_ts:1713924047770320, flushed_ts:1713924047778693, gen_to_freeze cost time:17, gen_to_submit cost time:249, submit_to_flush cost time:8373}, fs_cb_begin_ts=1713924048434828, log_life_time=664757) [2024-04-24 10:00:48.434967] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=109][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048258586000, v:0}, log_ts:{val:1713924048258586000, v:0}, lsn:{lsn:298516639}, submit_ts:1713924048258586}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:548720}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=176364, ctx_lock_wait_time=8) [2024-04-24 10:00:48.435042] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=73] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-04-24 10:00:48.435058] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=79][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924047767921000, v:0}, log_ts:{val:1713924047767921000, v:0}, lsn:{lsn:314128849}, submit_ts:1713924047767921}, this:0x14b15a14f340, is_inited_:true, trans_id:{txid:568567}, ls_id:{id:1}, ctx:0x14b15a14c3d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=667121, ctx_lock_wait_time=9) [2024-04-24 10:00:48.435570] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD281-0-0] [lt=91][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=304851, log_task={header:{begin_lsn:{lsn:79585532}, end_lsn:{lsn:79588418}, log_id:304851, min_scn:{val:1713924046872416001, v:0}, max_scn:{val:1713924047754365000, v:0}, data_len:2830, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:79580834}, data_checksum:1084423714, accum_checksum:2971793047, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924047755027, freeze_ts:1713924047760375, submit_ts:1713924047760497, flushed_ts:1713924047769006, gen_to_freeze cost time:5348, gen_to_submit cost time:5470, submit_to_flush cost time:8509}, fs_cb_begin_ts=1713924048435488, log_life_time=680461) [2024-04-24 10:00:48.435692] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=66][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67158973}], scn_array=[{val:1713924048276279001, v:0}], curr_size=122, accum_size=122, time_cost=33273) [2024-04-24 10:00:48.435774] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F295-0-0] [lt=90][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924047754365000, v:0}, log_ts:{val:1713924047754365000, v:0}, lsn:{lsn:79585785}, submit_ts:1713924047756374}, this:0x14b148006200, is_inited_:true, trans_id:{txid:327180}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=679379, ctx_lock_wait_time=11) [2024-04-24 10:00:48.435968] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2C198F24F-0-0] [lt=103][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:0, v:0}, log_ts:{val:1713924046872416001, v:0}, lsn:{lsn:79585588}, submit_ts:1713924047754365}, this:0x14b148005ae8, is_inited_:true, trans_id:{txid:327180}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:true, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=681585, ctx_lock_wait_time=10) [2024-04-24 10:00:48.439961] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=78] no need to update min/max version(new_version={val:1713924045694942002, v:0}, min_version={val:1713924045694942002, v:0}, max_version={val:1713924045694942002, v:0}) [2024-04-24 10:00:48.441156] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [862157][T1002_TeRec][T1001][YB420A000003-000616C2AE397390-0-0] [lt=168] get number(ret=0, service_type_=0, range=1, base_id=1713924048441114000, start_id=1713924048441114000, end_id=1713924048441114001) [2024-04-24 10:00:48.444593] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=81] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.444702] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861506][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=70] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:48.445002] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861507][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=117] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:48.445687] INFO [LIB] stat (utility.h:1197) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=150] [APPLY STAT CB APPEND COST TIME](cur_stat_count=11, stat_interval=5000000, avg (count/cost)=246, this=0x14b14711c280, extra_info="") [2024-04-24 10:00:48.445814] INFO [LIB] stat (utility.h:1197) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=103] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=11, stat_interval=5000000, avg (count/cost)=136861, this=0x14b14711c4b0, extra_info="") [2024-04-24 10:00:48.445987] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861508][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=111] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:48.445972] INFO [LIB] stat (utility.h:1197) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=142] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=11, stat_interval=5000000, avg (count/cost)=0, this=0x14b14711c6e0, extra_info="") [2024-04-24 10:00:48.446099] INFO [LIB] stat (utility.h:1197) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=112] [APPLY STAT CB EXECUTE TIME](cur_stat_count=11, stat_interval=5000000, avg (count/cost)=-700, this=0x14b14711c910, extra_info="") [2024-04-24 10:00:48.446212] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861509][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=79] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:48.446404] INFO [LIB] stat (utility.h:1197) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=88] [APPLY STAT CB TOTAL TIME](cur_stat_count=11, stat_interval=5000000, avg (count/cost)=136408, this=0x14b14711cb40, extra_info="") [2024-04-24 10:00:48.447591] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861511][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=75] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:48.447980] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861501][BatchIO][T0][Y0-0000000000000000-0-0] [lt=73] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:48.448260] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861500][BatchIO][T0][Y0-0000000000000000-0-0] [lt=74] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:48.449060] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [874648][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=112] ====== tenant freeze timer task ====== [2024-04-24 10:00:48.449060] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861502][BatchIO][T0][Y0-0000000000000000-0-0] [lt=76] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:48.449973] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861504][BatchIO][T0][Y0-0000000000000000-0-0] [lt=181] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:48.450023] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861503][BatchIO][T0][Y0-0000000000000000-0-0] [lt=84] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:48.450536] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2C208F1CE-0-0] [lt=161][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048271171000, v:0}, log_ts:{val:1713924048271171000, v:0}, lsn:{lsn:88118702}, submit_ts:1713924048271171}, this:0x14b147d6e480, is_inited_:true, trans_id:{txid:327186}, ls_id:{id:1001}, ctx:0x14b147d6b510, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=179340, ctx_lock_wait_time=14) [2024-04-24 10:00:48.450544] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2C198F24F-0-0] [lt=93][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048262374000, v:0}, log_ts:{val:1713924048262374000, v:0}, lsn:{lsn:88117638}, submit_ts:1713924048262374}, this:0x14b148367dc0, is_inited_:true, trans_id:{txid:327184}, ls_id:{id:1001}, ctx:0x14b148364e50, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=188151, ctx_lock_wait_time=11) [2024-04-24 10:00:48.450665] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F295-0-0] [lt=138][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048263644001, v:0}, log_ts:{val:1713924048263644001, v:0}, lsn:{lsn:88118185}, submit_ts:1713924048263644}, this:0x14b1481bc200, is_inited_:true, trans_id:{txid:327185}, ls_id:{id:1001}, ctx:0x14b1481b9290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=186966, ctx_lock_wait_time=7) [2024-04-24 10:00:48.450977] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861513][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=80] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:48.452761] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=129] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.454915] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2C0A8F1B4-0-0] [lt=81][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048297356000, v:0}, log_ts:{val:1713924048297356000, v:0}, lsn:{lsn:88119205}, submit_ts:1713924048297356}, this:0x14b1481b8b40, is_inited_:true, trans_id:{txid:327187}, ls_id:{id:1001}, ctx:0x14b1481b5bd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=157501, ctx_lock_wait_time=39) [2024-04-24 10:00:48.458114] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=73][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048268280000, v:0}, log_ts:{val:1713924048268280000, v:0}, lsn:{lsn:314130362}, submit_ts:1713924048269563}, this:0x14b15a130680, is_inited_:true, trans_id:{txid:568568}, ls_id:{id:1}, ctx:0x14b15a12d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=188539, ctx_lock_wait_time=6) [2024-04-24 10:00:48.458305] INFO [LIB] stat (utility.h:1197) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=116] [APPLY STAT CB APPEND COST TIME](cur_stat_count=17, stat_interval=5000000, avg (count/cost)=178, this=0x14b14d650280, extra_info="") [2024-04-24 10:00:48.458410] INFO [LIB] stat (utility.h:1197) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=88] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=17, stat_interval=5000000, avg (count/cost)=103426, this=0x14b14d6504b0, extra_info="") [2024-04-24 10:00:48.458575] INFO [LIB] stat (utility.h:1197) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=119] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=17, stat_interval=5000000, avg (count/cost)=0, this=0x14b14d6506e0, extra_info="") [2024-04-24 10:00:48.458709] INFO [LIB] stat (utility.h:1197) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=121] [APPLY STAT CB EXECUTE TIME](cur_stat_count=17, stat_interval=5000000, avg (count/cost)=-472, this=0x14b14d650910, extra_info="") [2024-04-24 10:00:48.458776] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=71][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048276279000, v:0}, log_ts:{val:1713924048276279000, v:0}, lsn:{lsn:1002188964}, submit_ts:1713924048276279}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877458}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=182474, ctx_lock_wait_time=6) [2024-04-24 10:00:48.458822] INFO [LIB] stat (utility.h:1197) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=88] [APPLY STAT CB TOTAL TIME](cur_stat_count=17, stat_interval=5000000, avg (count/cost)=103133, this=0x14b14d650b40, extra_info="") [2024-04-24 10:00:48.459005] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=97][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048250422000, v:0}, log_ts:{val:1713924048250422000, v:0}, lsn:{lsn:314129864}, submit_ts:1713924048250422}, this:0x14b15a160500, is_inited_:true, trans_id:{txid:568561}, ls_id:{id:1}, ctx:0x14b15a15d590, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=208557, ctx_lock_wait_time=16) [2024-04-24 10:00:48.459090] WDIAG [STORAGE.TRANS] get_tx_ctx (ob_trans_ctx_mgr_v4.cpp:1899) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E98-0-0] [lt=94][errcode=-6213] get transaction context error(ret=-6213, ret="OB_TRANS_CTX_NOT_EXIST", ls_id={id:1001}, tx_id={txid:877458}) [2024-04-24 10:00:48.459189] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1927) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E98-0-0] [lt=90][errcode=-6213] get coordinator tx context fail(ret=-6213, tx_id={txid:877458}, coord={id:1001}) [2024-04-24 10:00:48.459287] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E98-0-0] [lt=71][errcode=-6233] handle trans commit request failed(ret=-6233, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877458}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924048453040, cluster_id:1713874979}, expire_ts:1713924558320000, commit_start_scn:{val:1713924048246903000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:48.459417] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E98-0-0] [lt=113][errcode=-6233] handle txn message fail(ret=-6233, ret="OB_TRANS_COMMITED", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877458}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924048453040, cluster_id:1713874979}, expire_ts:1713924558320000, commit_start_scn:{val:1713924048246903000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:48.461792] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=92][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.463563] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [861713][T1_ReplaySrv5][T1][Y0-0000000000000000-0-0] [lt=82] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924036245286, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924048463546, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:48.466025] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:860) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=96] memtable handle rollback to successfuly(from_seq_no=1713924048464769, to_seq_no=1713924048461210, *this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877460} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=1, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}) [2024-04-24 10:00:48.467154] WDIAG [SERVER] process_curr_tenant (ob_information_query_response_time.cpp:131) [861898][T1_L0_G0][T1][YB420A000002-000616C2ACFD55A8-0-0] [lt=93][errcode=-4008] query response time size is 0(MTL_ID()=1, time=0x7ffd797dfed0, ret=-4008) [2024-04-24 10:00:48.467232] WDIAG [SERVER] process_curr_tenant (ob_information_query_response_time.cpp:131) [861898][T1_L0_G0][T1001][YB420A000002-000616C2ACFD55A8-0-0] [lt=69][errcode=-4008] query response time size is 0(MTL_ID()=1001, time=0x7ffd797dfed0, ret=-4008) [2024-04-24 10:00:48.467297] WDIAG [SERVER] process_curr_tenant (ob_information_query_response_time.cpp:131) [861898][T1_L0_G0][T1002][YB420A000002-000616C2ACFD55A8-0-0] [lt=58][errcode=-4008] query response time size is 0(MTL_ID()=1002, time=0x7ffd797dfed0, ret=-4008) [2024-04-24 10:00:48.467369] WDIAG [SERVER] process_curr_tenant (ob_information_query_response_time.cpp:131) [861898][T1_L0_G0][T1003][YB420A000002-000616C2ACFD55A8-0-0] [lt=65][errcode=-4008] query response time size is 0(MTL_ID()=1003, time=0x7ffd797dfed0, ret=-4008) [2024-04-24 10:00:48.467445] WDIAG [SERVER] process_curr_tenant (ob_information_query_response_time.cpp:131) [861898][T1_L0_G0][T1004][YB420A000002-000616C2ACFD55A8-0-0] [lt=65][errcode=-4008] query response time size is 0(MTL_ID()=1004, time=0x7ffd797dfed0, ret=-4008) [2024-04-24 10:00:48.467543] WDIAG [SERVER] inner_get_next_row (ob_information_query_response_time.cpp:105) [861898][T1_L0_G0][T1][YB420A000002-000616C2ACFD55A8-0-0] [lt=90][errcode=-4008] fail to execute(ret=-4008) [2024-04-24 10:00:48.469786] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:860) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BEEED977-0-0] [lt=133] memtable handle rollback to successfuly(from_seq_no=1713924048469483, to_seq_no=1713924048461211, *this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877461} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=1, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}) [2024-04-24 10:00:48.473134] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=152][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179741733}], scn_array=[{val:1713924048451070000, v:0}], curr_size=538, accum_size=4994, time_cost=10386) [2024-04-24 10:00:48.473093] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=62] tx gc loop thread is running(MTL_ID()=1004) [2024-04-24 10:00:48.473337] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=236] try gc retain ctx [2024-04-24 10:00:48.475136] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=136][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:88119596}], scn_array=[{val:1713924048463546000, v:0}], curr_size=447, accum_size=6931, time_cost=10838) [2024-04-24 10:00:48.476034] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [882037][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=99] get number(ret=0, service_type_=0, range=1, base_id=1713924048475556000, start_id=1713924048475556000, end_id=1713924048475556001) [2024-04-24 10:00:48.480085] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [2243678][T1002_L0_G0][T1002][YB420A000002-000616C2BEEED977-0-0] [lt=73] get number(ret=0, service_type_=0, range=1, base_id=1713924048479204000, start_id=1713924048479204000, end_id=1713924048479204001) [2024-04-24 10:00:48.480159] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=134] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1004, *ls_ptr={ls_meta:{tenant_id:1004, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895205050766000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924043670180000, v:0}, tablet_change_checkpoint_scn:{val:1713877445609286000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924058572929000, latest_log_ts:{val:1713924041908950001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713919200442801001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:5, palf_env_:0x14b176ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-04-24 10:00:48.483402] INFO [STORAGE.TRANS] print_retain_ctx_info (ob_tx_retain_ctx_mgr.cpp:273) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=99] [RetainCtxMgr] print retain ctx(ls_id={id:1}, this={retain_ctx_list_.get_size():295, max_wait_ckpt_ts_:{val:1713924045912367001, v:0}, last_push_gc_task_ts_:1713923772384693, skip_remove_cnt_:0}, retain_ctx_list_.get_first()={cause_:0, tx_id_:{txid:173250}, ls_id_:{id:1}, tx_ctx_:0x14b148289e50}, retain_ctx_list_.get_last()={cause_:0, tx_id_:{txid:176884}, ls_id_:{id:1}, tx_ctx_:0x14b04a660050}) [2024-04-24 10:00:48.487837] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [874706][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=101] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:769, free_obj_hold_cnt:74, allocator used:3351768, allocator total:3466624}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:72, free_obj_hold_cnt:32, allocator used:6821464, allocator total:6821464}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():154752, full allocator used:1600, full allocator total:220288}, tablets_mem=10508376, tablets_mem_limit=257698020, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:23, free_obj_hold_cnt:163, allocator used:369024, allocator total:392448}, pending_cnt=0, wait_gc_count=0, tablet count=841) [2024-04-24 10:00:48.488114] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=95] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1004, *ls_ptr={ls_meta:{tenant_id:1004, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1713895202258988000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924043670180000, v:0}, tablet_change_checkpoint_scn:{val:1713923988969639000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:7, palf_env_:0x14b176ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:2, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-04-24 10:00:48.489975] INFO [STORAGE.TRANS] print_retain_ctx_info (ob_tx_retain_ctx_mgr.cpp:273) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=95] [RetainCtxMgr] print retain ctx(ls_id={id:1001}, this={retain_ctx_list_.get_size():149, max_wait_ckpt_ts_:{val:1713924045471130001, v:0}, last_push_gc_task_ts_:1713923772386258, skip_remove_cnt_:0}, retain_ctx_list_.get_first()={cause_:0, tx_id_:{txid:173256}, ls_id_:{id:1001}, tx_ctx_:0x14b1482c4110}, retain_ctx_list_.get_last()={cause_:0, tx_id_:{txid:176884}, ls_id_:{id:1001}, tx_ctx_:0x14b055e16050}) [2024-04-24 10:00:48.490100] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=111] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.491340] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:213) [861671][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=94] start to check and clear obj lock(ls->get_ls_id()={id:1}) [2024-04-24 10:00:48.491359] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [874410][T1004_PalfGC][T1004][Y0-0000000000000000-0-0] [lt=110] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):33177, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):33177, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):390, sequence:1}) [2024-04-24 10:00:48.493699] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=62] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1004, *ls_ptr={ls_meta:{tenant_id:1004, ls_id:{id:1002}, ls_create_status:1, clog_checkpoint_scn:{val:1713923772377160001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924043670180000, v:0}, tablet_change_checkpoint_scn:{val:1713923978662610000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:2, proposal_id:0, palf_env_:0x14b176ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:2, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-04-24 10:00:48.495850] INFO [STORAGE.TRANS] print_retain_ctx_info (ob_tx_retain_ctx_mgr.cpp:273) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=123] [RetainCtxMgr] print retain ctx(ls_id={id:1002}, this={retain_ctx_list_.get_size():145, max_wait_ckpt_ts_:{val:1713924044495590001, v:0}, last_push_gc_task_ts_:1713923772387529, skip_remove_cnt_:0}, retain_ctx_list_.get_first()={cause_:0, tx_id_:{txid:173318}, ls_id_:{id:1002}, tx_ctx_:0x14b148391610}, retain_ctx_list_.get_last()={cause_:0, tx_id_:{txid:176882}, ls_id_:{id:1002}, tx_ctx_:0x14b147ec8590}) [2024-04-24 10:00:48.496004] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=136] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.501205] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59C8-0-0] [lt=132][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=299914, log_task={header:{begin_lsn:{lsn:40357918}, end_lsn:{lsn:40358367}, log_id:299914, min_scn:{val:1713924048269563000, v:0}, max_scn:{val:1713924048269563000, v:0}, data_len:393, proposal_id:5, prev_lsn:{lsn:40357212}, prev_proposal_id:5, committed_end_lsn:{lsn:40357918}, data_checksum:1363286283, accum_checksum:2741374451, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924048363605, freeze_ts:1713924048363612, submit_ts:1713924048363700, flushed_ts:1713924048376132, gen_to_freeze cost time:7, gen_to_submit cost time:95, submit_to_flush cost time:12432}, fs_cb_begin_ts=1713924048501149, log_life_time=137544) [2024-04-24 10:00:48.502514] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=98] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.502605] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=77] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.502702] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=88] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.503419] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874554][T1004_ApplySrv2][T1004][YB420A000003-000616C2BFC8F285-0-0] [lt=104][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048272353000, v:0}, log_ts:{val:1713924048272353000, v:0}, lsn:{lsn:79588596}, submit_ts:1713924048272353}, this:0x14b14895e040, is_inited_:true, trans_id:{txid:176909}, ls_id:{id:1}, ctx:0x14b14895b0d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=231051, ctx_lock_wait_time=8) [2024-04-24 10:00:48.509846] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=78] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8901173248, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:48.510049] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=163] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1572850, clean_num=31457) [2024-04-24 10:00:48.511785] WDIAG [STORAGE.TRANS] fill_redo_log (ob_memtable_context.cpp:655) [874589][T1004_L2_G0][T1004][YB420A000002-000616C2C34C0156-0-0] [lt=147][errcode=-5199] fill_redo_log fail(ret=-5199, trans_id="{txid:327195}", buf="mutaŒË³", buf_len=1964, buf_pos=0, pending_log_size=125, *this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465798592 trans_version={val:1713924048510522000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:327195} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=1 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=2 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}) [2024-04-24 10:00:48.511923] WDIAG [STORAGE.TRANS] fill_redo_log_ (ob_trans_part_ctx.cpp:2673) [874589][T1004_L2_G0][T1004][YB420A000002-000616C2C34C0156-0-0] [lt=132][errcode=-5199] fill redo log failed(ret=-5199, ret="OB_ERR_TOO_BIG_ROWSIZE", *this={this:0x14b148003290, ref:3, trans_id:{txid:327195}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924078510381, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048511427], ctx_create_time:1713924048510522}{ls_id:{id:1}, session_id:3221542242, part_trans_action:2, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79614837}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.3:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048509314, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:0}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b148006200, is_inited_:true, trans_id:{txid:327195}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:327195}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048510522000, v:0}, start_scn:{val:1713924048272353002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465798592 trans_version={val:1713924048510522000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:327195} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=1 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=2 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048272353002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048510522, block_frozen_memtable:null}) [2024-04-24 10:00:48.512073] WDIAG [STORAGE.TRANS] submit_redo_log_ (ob_trans_part_ctx.cpp:2765) [874589][T1004_L2_G0][T1004][YB420A000002-000616C2C34C0156-0-0] [lt=144][errcode=-5199] encounter too big row size error, need extend log buffer(ret=-5199, *this={this:0x14b148003290, ref:3, trans_id:{txid:327195}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924078510381, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048511427], ctx_create_time:1713924048510522}{ls_id:{id:1}, session_id:3221542242, part_trans_action:2, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79614837}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.3:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048509314, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:0}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b148006200, is_inited_:true, trans_id:{txid:327195}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:327195}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048510522000, v:0}, start_scn:{val:1713924048272353002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465798592 trans_version={val:1713924048510522000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:327195} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=1 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=2 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048272353002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048510522, block_frozen_memtable:null}) [2024-04-24 10:00:48.512247] INFO [STORAGE.TRANS] submit_redo_log_ (ob_trans_part_ctx.cpp:2779) [874589][T1004_L2_G0][T1004][YB420A000002-000616C2C34C0156-0-0] [lt=141] extend log buffer success(*this={this:0x14b148003290, ref:3, trans_id:{txid:327195}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924078510381, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048511427], ctx_create_time:1713924048510522}{ls_id:{id:1}, session_id:3221542242, part_trans_action:2, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79614837}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.3:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048509314, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:0}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b148006200, is_inited_:true, trans_id:{txid:327195}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:327195}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048510522000, v:0}, start_scn:{val:1713924048272353002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465798592 trans_version={val:1713924048510522000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:327195} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=1 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=2 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048272353002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048510522, block_frozen_memtable:null}, log_block={fill_buf_:{buf:0x14b05c204030, default_buf:0x14b0fd3ce798, len:1965056}, replay_buf_:null, len_:1965056, pos_:54, cur_log_type_:0, cb_arg_array_:[], big_segment_buf_:NULL}) [2024-04-24 10:00:48.512489] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [862171][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=78] ====== tenant freeze timer task ====== [2024-04-24 10:00:48.512655] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [862171][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=115] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=75497472, ctx.total_memstore_used_=109051904, ctx.total_memstore_hold_=109051904, memstore_freeze_trigger_limit_=91749580, tenant_id=1002) [2024-04-24 10:00:48.512773] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [862171][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=99] [TenantFreezer] oldest frozen memtable(list="") [2024-04-24 10:00:48.513395] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=191][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:40358367}], scn_array=[{val:1713924048484227000, v:0}], curr_size=122, accum_size=1726, time_cost=12140) [2024-04-24 10:00:48.514825] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:860) [1559406][T1003_L0_G0][T1003][YB420A000002-000616C2C30C01F0-0-0] [lt=107] memtable handle rollback to successfuly(from_seq_no=1713924048513062, to_seq_no=1713924048505717, *this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877385024160 max_table_version=1713877385024160 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338712} ls_id=1 callback_alloc_count=0 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=1, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}) [2024-04-24 10:00:48.517288] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:834) [874425][T1004_BalanceEx][T1004][YB420A000003-000616C2BB69A616-0-0] [lt=92] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2024-04-24 10:00:48.517407] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:171) [874425][T1004_BalanceEx][T1004][YB420A000003-000616C2BB69A616-0-0] [lt=111] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2024-04-24 10:00:48.519200] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=114][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67159095}], scn_array=[{val:1713924048479204000, v:0}], curr_size=122, accum_size=244, time_cost=16572) [2024-04-24 10:00:48.519272] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [861907][T1_L0_G0][T1][YB420A000002-000616C2AA6C607B-0-0] [lt=88] [RS_MGR] force set rs list(master_rs="10.0.0.2:2882") [2024-04-24 10:00:48.519372] INFO [SERVER] handle_heartbeat (ob_service.cpp:3094) [861907][T1_L0_G0][T1][YB420A000002-000616C2AA6C607B-0-0] [lt=84] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"10.0.0.3:2882", server_id:1, rs_addr:"10.0.0.2:2882", rs_server_status:1, epoch_id:5}, hb_response={zone:"zone1", server:"10.0.0.3:2882", sql_port:2881, build_version:"4.2.2.0_100010012024022719-c984fe7cb7a4cef85a40323a0d073f0c9b7b8235(Feb 27 2024 19:21:00)", start_service_time:1713875297913711, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=99) [2024-04-24 10:00:48.521119] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F72-0-0] [lt=129][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=482548, log_task={header:{begin_lsn:{lsn:298517051}, end_lsn:{lsn:298517692}, log_id:482548, min_scn:{val:1713924048367225000, v:0}, max_scn:{val:1713924048367225000, v:0}, data_len:585, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:298516583}, data_checksum:3052143852, accum_checksum:422322570, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924048367559, freeze_ts:1713924048367565, submit_ts:1713924048367736, flushed_ts:1713924048375918, gen_to_freeze cost time:6, gen_to_submit cost time:177, submit_to_flush cost time:8182}, fs_cb_begin_ts=1713924048521046, log_life_time=153487) [2024-04-24 10:00:48.521456] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=123][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048367225000, v:0}, log_ts:{val:1713924048367225000, v:0}, lsn:{lsn:298517107}, submit_ts:1713924048367225}, this:0x14b0fc0a5ac0, is_inited_:true, trans_id:{txid:548726}, ls_id:{id:1}, ctx:0x14b0fc0a2b50, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=154210, ctx_lock_wait_time=12) [2024-04-24 10:00:48.521946] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [874736][T1003_TenantWea][T1003][YB420A000003-000616C2B5FCA110-0-0] [lt=83] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924037110840, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924048521847, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:48.522329] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=138] get number(ret=0, service_type_=0, range=1, base_id=1713924048521847000, start_id=1713924048521847000, end_id=1713924048521847001) [2024-04-24 10:00:48.525083] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [861454][DumpLSLoc][T0][YB420A000003-000616C2AB0903FC-0-0] [lt=80] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713874979}, renew_time:1713924047294410, replica_locations:[{server:"10.0.0.2:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:5}, {server:"10.0.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2024-04-24 10:00:48.525125] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [874526][T1004_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=87] get number(ret=0, service_type_=0, range=1, base_id=1713924048524437000, start_id=1713924048524437000, end_id=1713924048524437001) [2024-04-24 10:00:48.525187] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [861454][DumpLSLoc][T0][YB420A000003-000616C2AB0903FC-0-0] [lt=88] [LS_LOCATION]dump tenant ls location caches(tenant_id=1001, tenant_ls_locations=[{cache_key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1713874979}, renew_time:1713924047302116, replica_locations:[{server:"10.0.0.2:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"10.0.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:5}]}]) [2024-04-24 10:00:48.525273] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [861454][DumpLSLoc][T0][YB420A000003-000616C2AB0903FC-0-0] [lt=79] [LS_LOCATION]dump tenant ls location caches(tenant_id=1002, tenant_ls_locations=[{cache_key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1713874979}, renew_time:1713924047304093, replica_locations:[{server:"10.0.0.2:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:5}, {server:"10.0.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1713874979}, renew_time:1713924047304081, replica_locations:[{server:"10.0.0.2:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"10.0.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:5}]}, {cache_key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1713874979}, renew_time:1713924047304087, replica_locations:[{server:"10.0.0.2:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"10.0.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:7}]}]) [2024-04-24 10:00:48.525376] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [861454][DumpLSLoc][T0][YB420A000003-000616C2AB0903FC-0-0] [lt=95] [LS_LOCATION]dump tenant ls location caches(tenant_id=1003, tenant_ls_locations=[{cache_key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1713874979}, renew_time:1713924047311801, replica_locations:[{server:"10.0.0.2:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"10.0.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:7}]}]) [2024-04-24 10:00:48.525519] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [861454][DumpLSLoc][T0][YB420A000003-000616C2AB0903FC-0-0] [lt=137] [LS_LOCATION]dump tenant ls location caches(tenant_id=1004, tenant_ls_locations=[{cache_key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1713874979}, renew_time:1713924047314270, replica_locations:[{server:"10.0.0.2:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"10.0.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:5}]}, {cache_key:{tenant_id:1004, ls_id:{id:1002}, cluster_id:1713874979}, renew_time:1713924047314284, replica_locations:[{server:"10.0.0.2:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:5}, {server:"10.0.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1001}, cluster_id:1713874979}, renew_time:1713924047314277, replica_locations:[{server:"10.0.0.2:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"10.0.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:7}]}]) [2024-04-24 10:00:48.526956] INFO [STORAGE.TRANS] print_statistics (ob_xa_define.cpp:467) [874522][T1004_ObXAHbWor][T1004][Y0-0000000000000000-0-0] [lt=112] xa statistics(total_active_xa_ctx_count=0, total_standby_clearup_count=0, total_success_xa_start=0, total_failure_xa_start=0, total_success_xa_prepare=0, total_failure_xa_prepare=0, total_success_xa_1pc_commit=0, total_failure_xa_1pc_commit=0, total_success_xa_2pc_commit=0, total_failure_xa_2pc_commit=0, total_failure_xa_rollback=0, total_success_dblink_promotion=0, total_failure_dblink_promotion=0, total_success_dblink=0, total_failure_dblink=0) [2024-04-24 10:00:48.527217] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=75][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.528232] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=73] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.533385] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2C0A8F1B4-0-0] [lt=105][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:0, v:0}, log_ts:{val:1713924048272353001, v:0}, lsn:{lsn:79589464}, submit_ts:1713924048432943}, this:0x14b1481c2868, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=100408, ctx_lock_wait_time=12) [2024-04-24 10:00:48.537517] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2C20C019D-0-0] [lt=92][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=301034, log_task={header:{begin_lsn:{lsn:165099312}, end_lsn:{lsn:165100885}, log_id:301034, min_scn:{val:1713924048374461000, v:0}, max_scn:{val:1713924048374461000, v:0}, data_len:1517, proposal_id:5, prev_lsn:{lsn:165099190}, prev_proposal_id:5, committed_end_lsn:{lsn:165099312}, data_checksum:173930186, accum_checksum:1092904725, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924048376364, freeze_ts:1713924048376375, submit_ts:1713924048376433, flushed_ts:1713924048383064, gen_to_freeze cost time:11, gen_to_submit cost time:69, submit_to_flush cost time:6631}, fs_cb_begin_ts=1713924048537439, log_life_time=161075) [2024-04-24 10:00:48.538752] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [862284][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=85] current clock is (clock_=0, max_seq_=0, mem_limit=213070636, cur_mem_hold=109051904, attr_.tenant_id_=1002) [2024-04-24 10:00:48.544783] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=123] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.556623] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:56) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2A8-0-0] [lt=220][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=16, len=885, pos=163, bitmap=NULL) [2024-04-24 10:00:48.557100] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:56) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2A8-0-0] [lt=153][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=16, len=325, pos=152, bitmap=NULL) [2024-04-24 10:00:48.557302] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [861709][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=154] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924036958043, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924048554997, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:48.557381] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:56) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2A8-0-0] [lt=74][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=11, len=184, pos=171, bitmap=NULL) [2024-04-24 10:00:48.557663] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:56) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2A8-0-0] [lt=97][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=16, len=241, pos=150, bitmap=NULL) [2024-04-24 10:00:48.558113] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:56) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2A8-0-0] [lt=88][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=7, len=136, pos=112, bitmap=NULL) [2024-04-24 10:00:48.562003] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=136][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.562472] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:860) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=164] memtable handle rollback to successfuly(from_seq_no=1713924048561701, to_seq_no=1713924048555040, *this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877463} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=1, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}) [2024-04-24 10:00:48.563069] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [862259][T1002_ReqMemEvi][T1002][Y0-0000000000000000-0-0] [lt=102] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=16106127, evict_high_size_level=90000, evict_batch_count=0, elapse_time=6, size_used=6629, mem_used=12478464) [2024-04-24 10:00:48.569599] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=144][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179743336}], scn_array=[{val:1713924048550885000, v:0}], curr_size=1028, accum_size=7087, time_cost=12879) [2024-04-24 10:00:48.573362] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=62] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.573487] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=116] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.574555] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [862430][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=86] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=90000, evict_batch_count=0, elapse_time=9, size_used=8474, mem_used=14558208) [2024-04-24 10:00:48.586145] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=228][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:1002193627}], scn_array=[{val:1713924048573312000, v:0}], curr_size=1573, accum_size=9671, time_cost=11000) [2024-04-24 10:00:48.587780] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [861669][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=138] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-04-24 10:00:48.589799] INFO [SHARE] deserialize_ (ob_rpc_struct.cpp:9227) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=82] deserialize txDesc from session(tx_desc={this:0x14b17297b4a0, tx_id:{txid:176888}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924045143051, op_sn:31, alloc_ts:-1, active_ts:1713924045142197, commit_ts:-1, finish_ts:-1, timeout_us:999996051, lock_timeout_us:-1, expire_ts:1713925045138248, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430286848638250, first_scn:1713924045154154, last_scn:1713924048586813, last_touch_ts:0}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, this={tenant_id:1004, tx_desc:{this:0x14b17297b4a0, tx_id:{txid:176888}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924045143051, op_sn:31, alloc_ts:-1, active_ts:1713924045142197, commit_ts:-1, finish_ts:-1, timeout_us:999996051, lock_timeout_us:-1, expire_ts:1713925045138248, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430286848638250, first_scn:1713924045154154, last_scn:1713924048586813, last_touch_ts:0}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, ls_id:{id:1001}, type:3, buf_.length():0x23c, request_id:1, register_flag:{need_flush_redo_instantly_:false, mds_base_scn_:{val:18446744073709551615, v:3}}}) [2024-04-24 10:00:48.590053] INFO [STORAGE.TRANS] statistics (ob_location_adapter.cpp:72) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=225] location adapter statistics(renew_access=0, total_access=353, error_count=0, renew_rate=0.000000000e+00) [2024-04-24 10:00:48.590698] INFO [STORAGE] init (ob_sstable.cpp:118) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=92] succeeded to init sstable(ret=0, this={ObITable:{this:0x14b0fd14fe20, key:{tablet_id:{id:200317}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1713895200302650000, v:0}}}, ref_cnt:0, upper_trans_version:1713895200302650000, timestamp:0}, this:0x14b0fd14fe20, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1713895200302650000, max_merged_trans_version:1713895200302650000, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:87, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:3, column_cnt_:22, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1713924046573560, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1713895200302650000, max_merged_trans_version_:1713895200302650000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:2, compressor_type:1, encrypt_id:0, master_key_id:-1, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x14b04a664280, column_checksum_count:22, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-04-24 10:00:48.590827] INFO [STORAGE] init (ob_medium_compaction_mgr.cpp:155) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=122] success to init medium clog recorder(ret=0, ls_id={id:1001}, tablet_id={id:200317}, max_saved_version=0) [2024-04-24 10:00:48.590967] INFO [STORAGE.TRANS] init_storage_recorder (ob_tablet_memtable_mgr.cpp:149) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=131] success to init storage recorder(ret=0, ls_id={id:1001}, tablet_id={id:200317}, max_saved_schema_version=1713924046573560, max_saved_medium_scn=0, compat_mode=0) [2024-04-24 10:00:48.591197] INFO [STORAGE] init_for_first_time_creation (ob_tablet.cpp:263) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=115] succeeded to init tablet for first time creation(ret=0, sstable={ObITable:{this:0x14b0fd14fe20, key:{tablet_id:{id:200317}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1713895200302650000, v:0}}}, ref_cnt:0, upper_trans_version:1713895200302650000, timestamp:0}, this:0x14b0fd14fe20, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1713895200302650000, max_merged_trans_version:1713895200302650000, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:87, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:3, column_cnt_:22, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1713924046573560, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1713895200302650000, max_merged_trans_version_:1713895200302650000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:2, compressor_type:1, encrypt_id:0, master_key_id:-1, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x14b04a664280, column_checksum_count:22, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}, *this={this:0x14b04a6edca0, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:2, length:0, tablet_addr:[0-0-0-3458765810900664320-0](offset=0,size=1208,type=3,seq=0), allocator:0x14b0e25e2970, tablet_meta:{version:1, ls_id:{id:1001}, tablet_id:{id:200317}, data_tablet_id:{id:200317}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1713895200302650000, multi_version_start:1713895200302650000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1713895200302650000, cur_report_version:1713895200302650000, data_checksum:0, row_count:0}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1713924046573560, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1713924046573560}, table_store_addr:{ptr:{this:0x14b04a6ee160, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:0}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), type:"MAJOR", tablet_id:{id:200317}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1713895200302650000, v:0}}, snapshot_version:1713895200302650000, max_merge_version:1713895200302650000}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{null}}}, addr:[0-0-0-3458764857417924608-0](offset=0,size=320,type=3,seq=0)}, storage_schema_addr:{ptr:{this:0x14b04a6ee630, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:0, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1713924046573560, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[{meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:1, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"DECIMAL UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"DECIMAL UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"DECIMAL UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"DECIMAL UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"DECIMAL UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"DECIMAL UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"DECIMAL UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"DATE", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}]}, addr:[0-0-0-3458764814468251648-0](offset=0,size=280,type=3,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:0}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:7, str:"MAX"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:0, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0}) [2024-04-24 10:00:48.591560] INFO [STORAGE.TRANS] create_tablet (ob_tablet_to_ls_cache.cpp:135) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=323] create tablet cache(ret=0, ret="OB_SUCCESS", tablet_id={id:200317}, ls_id={id:1001}) [2024-04-24 10:00:48.591992] INFO [MDS] register_to_mds_table_mgr (mds_table_mgr.cpp:100) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=116] register success(ret=0, ret="OB_SUCCESS", p_mds_table={this:0x14b0d6aad0b0, ls_id:{id:1001}, tablet_id:{id:200317}, flushing_scn:{val:18446744073709551615, v:3}, rec_scn:{val:4611686018427387903, v:0}, last_inner_recycled_scn:{val:0, v:0}, total_node_cnt:0, construct_sequence:12260, debug_info:{do_init_tablet_pointer:0x14b051fd4e88, do_remove_tablet_pointer:null, init_ts:"2024-04-24 10:00:48.591941", last_reset_ts:"1970-01-01 08:00:00.0", remove_ts:"1970-01-01 08:00:00.0", last_flush_ts:"1970-01-01 08:00:00.0", switch_to_empty_shell_ts:"1970-01-01 08:00:00.0", init_trace_id:YB420A000002-000616C2C30C01F0-0-0, remove_trace_id:Y0-0000000000000000-0-0}}) [2024-04-24 10:00:48.592101] INFO [MDS] init (mds_table_base.cpp:76) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=97] mds table inited(ret=0, ret="OB_SUCCESS", *this={this:0x14b0d6aad0b0, ls_id:{id:1001}, tablet_id:{id:200317}, flushing_scn:{val:18446744073709551615, v:3}, rec_scn:{val:4611686018427387903, v:0}, last_inner_recycled_scn:{val:0, v:0}, total_node_cnt:0, construct_sequence:12260, debug_info:{do_init_tablet_pointer:0x14b051fd4e88, do_remove_tablet_pointer:null, init_ts:"2024-04-24 10:00:48.591941", last_reset_ts:"1970-01-01 08:00:00.0", remove_ts:"1970-01-01 08:00:00.0", last_flush_ts:"1970-01-01 08:00:00.0", switch_to_empty_shell_ts:"1970-01-01 08:00:00.0", init_trace_id:YB420A000002-000616C2C30C01F0-0-0, remove_trace_id:Y0-0000000000000000-0-0}}) [2024-04-24 10:00:48.592468] INFO [MDS.EVENT]construct_insert_record_user_mds_node_ (mds_row.ipp:216) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=74] WRITE_NODE(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:200317}}, event={alloc:null, timestamp:"2024-04-24 10:00:48.592454", event:"WRITE_NODE", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176888, seq_no:0, redo_scn:{val:4611686018427387903, v:0}, end_scn:{val:4611686018427387903, v:0}, trans_version:{val:4611686018427387903, v:0}, node_type:1, state:0}) [2024-04-24 10:00:48.592659] INFO [STORAGE] set_tablet_status (ob_ls_tablet_service.cpp:2382) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=106] succeeded to set tablet status(ret=0, tablet_id={id:200317}, tablet_status={tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}) [2024-04-24 10:00:48.593121] INFO [STORAGE] init (ob_sstable.cpp:118) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=120] succeeded to init sstable(ret=0, this={ObITable:{this:0x14b0fd14fe20, key:{tablet_id:{id:1152921504606847581}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1713895200302650000, v:0}}}, ref_cnt:0, upper_trans_version:1713895200302650000, timestamp:0}, this:0x14b0fd14fe20, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1713895200302650000, max_merged_trans_version:1713895200302650000, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:87, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:2, rowkey_column_count_:6, column_cnt_:7, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1713924046573568, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1713895200302650000, max_merged_trans_version_:1713895200302650000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:2, compressor_type:1, encrypt_id:0, master_key_id:-1, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x14b0617fa280, column_checksum_count:7, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-04-24 10:00:48.593269] INFO [STORAGE] init (ob_medium_compaction_mgr.cpp:155) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=140] success to init medium clog recorder(ret=0, ls_id={id:1001}, tablet_id={id:1152921504606847581}, max_saved_version=0) [2024-04-24 10:00:48.593336] INFO [STORAGE.TRANS] init_storage_recorder (ob_tablet_memtable_mgr.cpp:149) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=58] success to init storage recorder(ret=0, ls_id={id:1001}, tablet_id={id:1152921504606847581}, max_saved_schema_version=1713924046573568, max_saved_medium_scn=0, compat_mode=0) [2024-04-24 10:00:48.593492] INFO [STORAGE] init_for_first_time_creation (ob_tablet.cpp:263) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=83] succeeded to init tablet for first time creation(ret=0, sstable={ObITable:{this:0x14b0fd14fe20, key:{tablet_id:{id:1152921504606847581}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1713895200302650000, v:0}}}, ref_cnt:0, upper_trans_version:1713895200302650000, timestamp:0}, this:0x14b0fd14fe20, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1713895200302650000, max_merged_trans_version:1713895200302650000, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:87, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:2, rowkey_column_count_:6, column_cnt_:7, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1713924046573568, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1713895200302650000, max_merged_trans_version_:1713895200302650000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:2, compressor_type:1, encrypt_id:0, master_key_id:-1, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x14b0617fa280, column_checksum_count:7, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}, *this={this:0x14b04a6efbe0, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:2, length:0, tablet_addr:[0-0-0-3458765810900664320-0](offset=0,size=1208,type=3,seq=0), allocator:0x14b0e25e2a20, tablet_meta:{version:1, ls_id:{id:1001}, tablet_id:{id:1152921504606847581}, data_tablet_id:{id:200317}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1713895200302650000, multi_version_start:1713895200302650000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1713895200302650000, cur_report_version:1713895200302650000, data_checksum:0, row_count:0}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1713924046573568, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1713924046573568}, table_store_addr:{ptr:{this:0x14b04a6f00a0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:0}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), type:"MAJOR", tablet_id:{id:1152921504606847581}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1713895200302650000, v:0}}, snapshot_version:1713895200302650000, max_merge_version:1713895200302650000}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{null}}}, addr:[0-0-0-3458764857417924608-0](offset=0,size=320,type=3,seq=0)}, storage_schema_addr:{ptr:{this:0x14b04a6f04f0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:0, compat_mode:0, table_type:5, index_type:2, index_status:1, row_store_type:1, schema_version:1713924046573568, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:17, meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"DATE", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[{meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:1, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:1, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"DATE", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:1, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:1, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}]}, addr:[0-0-0-3458764814468251648-0](offset=0,size=280,type=3,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:0}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:7, str:"MAX"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:0, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0}) [2024-04-24 10:00:48.593758] INFO [STORAGE.TRANS] create_tablet (ob_tablet_to_ls_cache.cpp:135) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=211] create tablet cache(ret=0, ret="OB_SUCCESS", tablet_id={id:1152921504606847581}, ls_id={id:1001}) [2024-04-24 10:00:48.594038] INFO [MDS] register_to_mds_table_mgr (mds_table_mgr.cpp:100) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=67] register success(ret=0, ret="OB_SUCCESS", p_mds_table={this:0x14b0d6aad480, ls_id:{id:1001}, tablet_id:{id:1152921504606847581}, flushing_scn:{val:18446744073709551615, v:3}, rec_scn:{val:4611686018427387903, v:0}, last_inner_recycled_scn:{val:0, v:0}, total_node_cnt:0, construct_sequence:12261, debug_info:{do_init_tablet_pointer:0x14b051fd4fd8, do_remove_tablet_pointer:null, init_ts:"2024-04-24 10:00:48.594006", last_reset_ts:"1970-01-01 08:00:00.0", remove_ts:"1970-01-01 08:00:00.0", last_flush_ts:"1970-01-01 08:00:00.0", switch_to_empty_shell_ts:"1970-01-01 08:00:00.0", init_trace_id:YB420A000002-000616C2C30C01F0-0-0, remove_trace_id:Y0-0000000000000000-0-0}}) [2024-04-24 10:00:48.594184] INFO [MDS] init (mds_table_base.cpp:76) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=118] mds table inited(ret=0, ret="OB_SUCCESS", *this={this:0x14b0d6aad480, ls_id:{id:1001}, tablet_id:{id:1152921504606847581}, flushing_scn:{val:18446744073709551615, v:3}, rec_scn:{val:4611686018427387903, v:0}, last_inner_recycled_scn:{val:0, v:0}, total_node_cnt:0, construct_sequence:12261, debug_info:{do_init_tablet_pointer:0x14b051fd4fd8, do_remove_tablet_pointer:null, init_ts:"2024-04-24 10:00:48.594006", last_reset_ts:"1970-01-01 08:00:00.0", remove_ts:"1970-01-01 08:00:00.0", last_flush_ts:"1970-01-01 08:00:00.0", switch_to_empty_shell_ts:"1970-01-01 08:00:00.0", init_trace_id:YB420A000002-000616C2C30C01F0-0-0, remove_trace_id:Y0-0000000000000000-0-0}}) [2024-04-24 10:00:48.594459] INFO [MDS.EVENT]construct_insert_record_user_mds_node_ (mds_row.ipp:216) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=61] WRITE_NODE(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:1152921504606847581}}, event={alloc:null, timestamp:"2024-04-24 10:00:48.594437", event:"WRITE_NODE", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176888, seq_no:0, redo_scn:{val:4611686018427387903, v:0}, end_scn:{val:4611686018427387903, v:0}, trans_version:{val:4611686018427387903, v:0}, node_type:1, state:0}) [2024-04-24 10:00:48.594677] INFO [STORAGE] set_tablet_status (ob_ls_tablet_service.cpp:2382) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=131] succeeded to set tablet status(ret=0, tablet_id={id:1152921504606847581}, tablet_status={tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}) [2024-04-24 10:00:48.594864] INFO [STORAGE] set_tablet_gc_trigger (ob_tablet_gc_service.cpp:277) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=94] set_tablet_gc_trigger(this={tablet_persist_trigger:2, is_inited:true}) [2024-04-24 10:00:48.594993] INFO [MDS] register_process (ob_tablet_create_mds_helper.cpp:84) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=118] create tablet register(ret=0, ret="OB_SUCCESS", arg={ls_id:{id:1001}, major_frozen_scn:{val:1713895200302650000, v:0}, total_tablet_cnt:2, tablets:{ [874592] [YB420A000002-000616C2C30C01F0-0-0] data_tablet_id:{id:200317}, tablet_ids:[{id:200317}, {id:1152921504606847581}], compat_mode:0, is_create_bind_hidden_tablets:false }}) [2024-04-24 10:00:48.594993] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [861423][TbltTblUp0][T1003][YB420A000003-000616C2AB2901F3-0-0] [lt=94] get number(ret=0, service_type_=0, range=1, base_id=1713924048592701000, start_id=1713924048592701000, end_id=1713924048592701001) [2024-04-24 10:00:48.595096] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [861744][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=75] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1713924044295272, tenant_id:1004, sequence_id:5497558139157}) [2024-04-24 10:00:48.595103] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=89] notify MDS(ret=0, trans_id_={txid:176888}, ls_id_={id:1001}, notify_type=0, log_ts={val:18446744073709551615, v:3}, notify_array.count()=1, notify_array=[{register_no_:0, has_submitted_:false, has_synced_:false, type:3, data_.length():572}], total_time=0) [2024-04-24 10:00:48.595202] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [861744][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=94] try reload schema success(schema_version=1713878795542168, refresh_schema_info={schema_version:1713924044295272, tenant_id:1004, sequence_id:5497558139157}, schema_ret=0) [2024-04-24 10:00:48.595221] INFO [STORAGE.TRANS] register_mds_into_tx (ob_trans_service.cpp:912) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=99] register multi data source result(ret=0, ret="OB_SUCCESS", arg={tenant_id:1004, tx_desc:{this:0x14b17297b4a0, tx_id:{txid:176888}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924045143051, op_sn:31, alloc_ts:-1, active_ts:1713924045142197, commit_ts:-1, finish_ts:-1, timeout_us:999996051, lock_timeout_us:-1, expire_ts:1713925045138248, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430286848638250, first_scn:1713924045154154, last_scn:1713924048586813, last_touch_ts:0}, {id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993240235991579, first_scn:1713924048589224, last_scn:1713924048589224, last_touch_ts:1713924048592705}], exec_info_reap_ts:1713924048592704, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, ls_id:{id:1001}, type:3, buf_.length():0x23c, request_id:1, register_flag:{need_flush_redo_instantly_:false, mds_base_scn_:{val:18446744073709551615, v:3}}}, result={result:0, tx_result:{incomplete:false, parts:[{id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993240235991579, first_scn:1713924048589224, last_scn:1713924048589224, last_touch_ts:1713924048592704}], touched_ls_list:[], cflict_txs:[]}}, tx_desc={this:0x14b17297b4a0, tx_id:{txid:176888}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924045143051, op_sn:31, alloc_ts:-1, active_ts:1713924045142197, commit_ts:-1, finish_ts:-1, timeout_us:999996051, lock_timeout_us:-1, expire_ts:1713925045138248, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430286848638250, first_scn:1713924045154154, last_scn:1713924048586813, last_touch_ts:0}, {id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993240235991579, first_scn:1713924048589224, last_scn:1713924048589224, last_touch_ts:1713924048592705}], exec_info_reap_ts:1713924048592704, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, local_retry_cnt=0, retry_cnt=0, request_id=1, time_guard=time guard have no click events for optimization) [2024-04-24 10:00:48.595262] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92486-0-0] [lt=56] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1713924044295272, tenant_id:1004, sequence_id:5497558139157}) [2024-04-24 10:00:48.595367] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:280) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92486-0-0] [lt=88] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1713924044295272, tenant_id:1004, sequence_id:5497558139157}, schema_info={schema_version:1713924044295272, tenant_id:1004, sequence_id:5497558139157}) [2024-04-24 10:00:48.595394] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [861423][TbltTblUp0][T1003][YB420A000003-000616C2AB2901F3-0-0] [lt=89] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924038322667, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924048592701, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:48.602073] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59C9-0-0] [lt=130][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=299915, log_task={header:{begin_lsn:{lsn:40358367}, end_lsn:{lsn:40358489}, log_id:299915, min_scn:{val:1713924048484227000, v:0}, max_scn:{val:1713924048484227000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:40357918}, prev_proposal_id:5, committed_end_lsn:{lsn:40358367}, data_checksum:1715074492, accum_checksum:1802938019, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924048501057, freeze_ts:1713924048501066, submit_ts:1713924048501116, flushed_ts:1713924048513620, gen_to_freeze cost time:9, gen_to_submit cost time:59, submit_to_flush cost time:12504}, fs_cb_begin_ts=1713924048602009, log_life_time=100952) [2024-04-24 10:00:48.602470] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=140] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.602604] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=114] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.602680] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=70] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.611235] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [861686][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=136] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-24 10:00:48.616039] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [874613][T1003_Occam][T1003][YB420A000003-000616C2B70982E6-0-0] [lt=115] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:48.615976] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=60][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:298519438}], scn_array=[{val:1713924048596957000, v:0}], curr_size=973, accum_size=7075, time_cost=18197) [2024-04-24 10:00:48.618077] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=184][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179744364}], scn_array=[{val:1713924048594493000, v:0}], curr_size=1072, accum_size=8159, time_cost=17075) [2024-04-24 10:00:48.618326] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=121][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:40358489}], scn_array=[{val:1713924048598120000, v:0}], curr_size=122, accum_size=1848, time_cost=16206) [2024-04-24 10:00:48.619741] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [874532][T1004_LSArchive][T1004][YB420A000003-000616C2B568F0E7-0-0] [lt=149] gc stale ls task succ [2024-04-24 10:00:48.627446] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=76][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.635404] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=71] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-04-24 10:00:48.638842] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [875192][T1004_L0_G5][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=166] get number(ret=0, service_type_=0, range=1, base_id=1713924048638209000, start_id=1713924048638209000, end_id=1713924048638209001) [2024-04-24 10:00:48.640917] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [861673][T1_LSMetaCh][T1][YB420A000003-000616C2A909ADA2-0-0] [lt=131] fetch root ls success(local_cluster_id=1713874979, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.2:2882", sql_port:2881, role:1, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:5, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713921656566585, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.3:2882", sql_port:2881, role:2, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["10.0.0.2:2882", "10.0.0.3:2882"]) [2024-04-24 10:00:48.641996] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [862431][T1001_TenantWea][T1001][YB420A000003-000616C2AEECE7EB-0-0] [lt=122] current clock is (clock_=0, max_seq_=0, mem_limit=192099116, cur_mem_hold=130023424, attr_.tenant_id_=1001) [2024-04-24 10:00:48.642320] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=124] get number(ret=0, service_type_=0, range=1, base_id=1713924048641803000, start_id=1713924048641803000, end_id=1713924048641803001) [2024-04-24 10:00:48.643750] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2648) [861573][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=182] [PALF_DUMP](palf_id=1, self="10.0.0.3:2882", [SlidingWindow]={palf_id:1, self:"10.0.0.3:2882", lsn_allocator:{max_log_id:364241, max_lsn:179745436, max_scn:{val:1713924048594493000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:179745436}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:137802396}}, last_submit_lsn:{lsn:179744364}, last_submit_end_lsn:{lsn:179745436}, last_submit_log_id:364241, last_submit_log_pid:5, max_flushed_lsn:{lsn:179744364}, max_flushed_end_lsn:{lsn:179745436}, max_flushed_log_pid:5, committed_end_lsn:{lsn:179744364}, last_slide_log_id:364240, last_slide_scn:{val:1713924048550885000, v:0}, last_slide_lsn:{lsn:179743336}, last_slide_end_lsn:{lsn:179744364}, last_slide_log_pid:5, last_slide_log_accum_checksum:1786168960, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x14b186b46530}, [StateMgr]={this:0x14b186b4a2b0, self:"10.0.0.3:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:5}, leader:"10.0.0.2:2882", leader_epoch:8, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:364241, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x14b186b50b70, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"10.0.0.3:2882", alive_paxos_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, alive_paxos_replica_num:2, log_ms_meta:{version:3, proposal_id:5, prev:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:5, config_seq:6}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:4, prev_lsn:{lsn:170356591}, prev_mode_pid:5}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:5, config_seq:6}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1713924048142522, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1713924048142522, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x14b186b47c70}, [ModeMgr]={palf_id:1, self:"10.0.0.3:2882", applied_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:67104768}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:5}, log_config_meta:{version:3, proposal_id:5, prev:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:5, config_seq:6}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:4, prev_lsn:{lsn:170356591}, prev_mode_pid:5}, log_snapshot_meta:{version:1, base_lsn:{lsn:67104768}, prev_log_info:{log_id:-1, lsn:{lsn:18446744073709551615}, scn:{val:18446744073709551615, v:3}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:73728}, readable_log_tail:{lsn:73728}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:201314304}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/data/admin/test/oceanbase/store/clog/tenant_1/1/meta", dir_fd:144, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67031040, block_header_serialize_buf_:0x14b186b4c438, flashback_version:0}, log_storage:{log_tail:{lsn:179745436}, readable_log_tail:{lsn:179745436}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:1713908680224277001, v:0}, curr_block_id:2, palf_id:1}, block_mgr:{log_dir:"/data/admin/test/oceanbase/store/clog/tenant_1/1/log", dir_fd:145, min_block_id:0, max_block_id:3, curr_writable_block_id:2}, logical_block_size_:67104768, curr_block_writable_size_:21568868, block_header_serialize_buf_:0x14b186b4e0b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1713921656244433, this:0x14b186b4a9f0}, [Reconfirm]={palf_id:1, self:"10.0.0.3:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x14b186b4a430}) [2024-04-24 10:00:48.644131] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2742) [861573][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=372][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="10.0.0.3:2882", tmp_server={server:"10.0.0.2:2882", timestamp:1, flag:0}) [2024-04-24 10:00:48.644297] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2652) [861573][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=153] [PALF_DUMP](palf_id=1, self="10.0.0.3:2882", ack_info_list=[]) [2024-04-24 10:00:48.644517] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2742) [861573][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=189][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="10.0.0.3:2882", tmp_server={server:"10.0.0.2:2882", timestamp:1, flag:0}) [2024-04-24 10:00:48.644737] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [861685][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=334] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=15, size_used=15931, mem_used=27036672) [2024-04-24 10:00:48.644677] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4290) [861573][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=137] current log_task status(palf_id=1, self="10.0.0.3:2882", log_id=364241, log_task={header:{begin_lsn:{lsn:179744364}, end_lsn:{lsn:179745436}, log_id:364241, min_scn:{val:1713924048594493000, v:0}, max_scn:{val:1713924048594493000, v:0}, data_len:1016, proposal_id:5, prev_lsn:{lsn:179743336}, prev_proposal_id:5, committed_end_lsn:{lsn:179744364}, data_checksum:177810747, accum_checksum:1038924345, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713924048600752, freeze_ts:1713924048600761, submit_ts:1713924048600823, flushed_ts:1713924048618365, gen_to_freeze cost time:9, gen_to_submit cost time:71, submit_to_flush cost time:17542}, ack_info_list=[]) [2024-04-24 10:00:48.645025] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=124] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.648781] INFO [SHARE] deserialize_ (ob_rpc_struct.cpp:9227) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=129] deserialize txDesc from session(tx_desc={this:0x14b1729ab6e0, tx_id:{txid:176888}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924045143051, op_sn:34, alloc_ts:-1, active_ts:1713924045142197, commit_ts:-1, finish_ts:-1, timeout_us:999996051, lock_timeout_us:-1, expire_ts:1713925045138248, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430286848638250, first_scn:1713924045154154, last_scn:1713924048600684, last_touch_ts:0}, {id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993240235991579, first_scn:1713924048589224, last_scn:1713924048589224, last_touch_ts:0}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, this={tenant_id:1004, tx_desc:{this:0x14b1729ab6e0, tx_id:{txid:176888}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924045143051, op_sn:34, alloc_ts:-1, active_ts:1713924045142197, commit_ts:-1, finish_ts:-1, timeout_us:999996051, lock_timeout_us:-1, expire_ts:1713925045138248, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430286848638250, first_scn:1713924045154154, last_scn:1713924048600684, last_touch_ts:0}, {id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993240235991579, first_scn:1713924048589224, last_scn:1713924048589224, last_touch_ts:0}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, ls_id:{id:1}, type:6, buf_.length():0x521, request_id:1, register_flag:{need_flush_redo_instantly_:false, mds_base_scn_:{val:18446744073709551615, v:3}}}) [2024-04-24 10:00:48.649093] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=212] notify MDS(ret=0, trans_id_={txid:176888}, ls_id_={id:1}, notify_type=0, log_ts={val:18446744073709551615, v:3}, notify_array.count()=1, notify_array=[{register_no_:0, has_submitted_:false, has_synced_:false, type:6, data_.length():1313}], total_time=0) [2024-04-24 10:00:48.649211] INFO [STORAGE.TRANS] register_mds_into_tx (ob_trans_service.cpp:912) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=103] register multi data source result(ret=0, ret="OB_SUCCESS", arg={tenant_id:1004, tx_desc:{this:0x14b1729ab6e0, tx_id:{txid:176888}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924045143051, op_sn:34, alloc_ts:-1, active_ts:1713924045142197, commit_ts:-1, finish_ts:-1, timeout_us:999996051, lock_timeout_us:-1, expire_ts:1713925045138248, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430286848638250, first_scn:1713924045154154, last_scn:1713924048648217, last_touch_ts:1713924048648219}, {id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993240235991579, first_scn:1713924048589224, last_scn:1713924048589224, last_touch_ts:0}], exec_info_reap_ts:1713924048648218, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, ls_id:{id:1}, type:6, buf_.length():0x521, request_id:1, register_flag:{need_flush_redo_instantly_:false, mds_base_scn_:{val:18446744073709551615, v:3}}}, result={result:0, tx_result:{incomplete:false, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430286848638250, first_scn:1713924045154154, last_scn:1713924048648217, last_touch_ts:1713924048648218}], touched_ls_list:[], cflict_txs:[]}}, tx_desc={this:0x14b1729ab6e0, tx_id:{txid:176888}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924045143051, op_sn:34, alloc_ts:-1, active_ts:1713924045142197, commit_ts:-1, finish_ts:-1, timeout_us:999996051, lock_timeout_us:-1, expire_ts:1713925045138248, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430286848638250, first_scn:1713924045154154, last_scn:1713924048648217, last_touch_ts:1713924048648219}, {id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993240235991579, first_scn:1713924048589224, last_scn:1713924048589224, last_touch_ts:0}], exec_info_reap_ts:1713924048648218, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, local_retry_cnt=0, retry_cnt=0, request_id=1, time_guard=time guard have no click events for optimization) [2024-04-24 10:00:48.653157] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=136] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.656667] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [874617][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=98] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2024-04-24 10:00:48.657028] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=82] get number(ret=0, service_type_=0, range=1, base_id=1713924048656929000, start_id=1713924048656929000, end_id=1713924048656929001) [2024-04-24 10:00:48.662212] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=75][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.665246] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=74] notify MDS(ret=0, trans_id_={txid:176888}, ls_id_={id:1}, notify_type=2, log_ts={val:18446744073709551615, v:3}, notify_array.count()=1, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:false, type:6, data_.length():1313}], total_time=0) [2024-04-24 10:00:48.666553] INFO [STORAGE.TRANS] submit_multi_data_source_ (ob_trans_part_ctx.cpp:6456) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=99] submit MDS redo with barrier or base_scn successfully(ret=0, trans_id_={txid:176888}, ls_id_={id:1001}, log_cb={ObTxBaseLogCb:{base_ts:{val:0, v:0}, log_ts:{val:1713924048588043001, v:0}, lsn:{lsn:88120890}, submit_ts:1713924048665345}, this:0x14b055e208a8, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():1, range_array_:[{register_no_:1, has_submitted_:true, has_synced_:false, type:3, data_.length():572}]}, cb_arg_array_:[{log_type:0x4, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, mds_cache_={unsubmitted_size_:0, mds_list_.size():1, max_register_no_:1}, exec_info_.multi_data_source_=[], mds_base_scn={val:0, v:0}, barrier_type=2) [2024-04-24 10:00:48.666944] INFO [MDS.EVENT]before_prepare_ (mds_node.ipp:174) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=106] BEFORE_PREPARE(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:200317}}, event={alloc:null, timestamp:"2024-04-24 10:00:48.666809", event:"BEFORE_PREPARE", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176888, seq_no:0, redo_scn:{val:4611686018427387903, v:0}, end_scn:{val:4611686018427387903, v:0}, trans_version:{val:0, v:0}, node_type:1, state:1}) [2024-04-24 10:00:48.667167] INFO [MDS.EVENT]before_prepare_ (mds_node.ipp:174) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=151] BEFORE_PREPARE(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:1152921504606847581}}, event={alloc:null, timestamp:"2024-04-24 10:00:48.667145", event:"BEFORE_PREPARE", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176888, seq_no:0, redo_scn:{val:4611686018427387903, v:0}, end_scn:{val:4611686018427387903, v:0}, trans_version:{val:0, v:0}, node_type:1, state:1}) [2024-04-24 10:00:48.667256] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=70] notify MDS(ret=0, trans_id_={txid:176888}, ls_id_={id:1001}, notify_type=2, log_ts={val:18446744073709551615, v:3}, notify_array.count()=1, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:false, type:3, data_.length():572}], total_time=0) [2024-04-24 10:00:48.667496] INFO [RPC] process (ob_batch_processor.cpp:106) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2C30C01F0-0-0] [lt=74] batch rpc statistics(clog_batch_nodelay_cnt=0, clog_batch_cnt=0, trx_batch_cnt=1, sql_batch_cnt=0) [2024-04-24 10:00:48.670476] INFO [SQL.JO] compute_table_location (ob_join_order.cpp:278) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1E97-0-0] [lt=100] succeed to calculate base table sharding info(table_id=18446744073709551614, ref_table_id=104, is_global_index=false) [2024-04-24 10:00:48.670608] INFO [SQL.JO] init_est_sel_info_for_access_path (ob_join_order.cpp:12189) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1E97-0-0] [lt=101] total rowcount, use statistics(table_meta_info_.table_row_count_=10, table_meta_info_.average_row_size_=2.060000000000000000e+02, table_meta_info_.micro_block_count_=1, table_meta_info_.part_size_=2.060000000000000000e+03) [2024-04-24 10:00:48.673687] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=108] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.673866] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=143] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.674008] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=136] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.690952] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2648) [874414][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=63] [PALF_DUMP](palf_id=1, self="10.0.0.3:2882", [SlidingWindow]={palf_id:1, self:"10.0.0.3:2882", lsn_allocator:{max_log_id:304859, max_lsn:79624288, max_scn:{val:1713924048664122000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:79624288}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:37681248}}, last_submit_lsn:{lsn:79620139}, last_submit_end_lsn:{lsn:79624288}, last_submit_log_id:304859, last_submit_log_pid:5, max_flushed_lsn:{lsn:79620139}, max_flushed_end_lsn:{lsn:79624288}, max_flushed_log_pid:5, committed_end_lsn:{lsn:79617788}, last_slide_log_id:304856, last_slide_scn:{val:1713924048510522000, v:0}, last_slide_lsn:{lsn:79615034}, last_slide_end_lsn:{lsn:79617788}, last_slide_log_pid:5, last_slide_log_accum_checksum:2334418479, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x14b0efb28530}, [StateMgr]={this:0x14b0efb2c2b0, self:"10.0.0.3:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:5}, leader:"10.0.0.3:2882", leader_epoch:8, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:69192977}, scan_disk_log_finished:true, last_check_start_id:304857, is_changing_config_with_arb:false, reconfirm_start_time_us:1713921656085382, palf_role_change_cb:0x14b0efb32b70, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"10.0.0.3:2882", alive_paxos_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, alive_paxos_replica_num:2, log_ms_meta:{version:3, proposal_id:5, prev:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:5, config_seq:6}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:4, prev_lsn:{lsn:69192479}, prev_mode_pid:5}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:4, prev_lsn:{lsn:69192479}, prev_end_lsn:{lsn:69192977}, prev_mode_pid:5}, reconfig_barrier:{prev_log_proposal_id:4, prev_lsn:{lsn:69192479}, prev_end_lsn:{lsn:69192977}, prev_mode_pid:5}, persistent_config_version:{proposal_id:5, config_seq:6}, ms_ack_list:0, resend_config_version:{proposal_id:5, config_seq:6}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x14b0efb29c70}, [ModeMgr]={palf_id:1, self:"10.0.0.3:2882", applied_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1713921656487211, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:5}, log_config_meta:{version:3, proposal_id:5, prev:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:5, config_seq:6}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:4, prev_lsn:{lsn:69192479}, prev_mode_pid:5}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:69632}, readable_log_tail:{lsn:69632}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/data/admin/test/oceanbase/store/clog/tenant_1004/1/meta", dir_fd:193, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67035136, block_header_serialize_buf_:0x14b0efb2e438, flashback_version:0}, log_storage:{log_tail:{lsn:79624288}, readable_log_tail:{lsn:79624288}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1713919494579765001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/data/admin/test/oceanbase/store/clog/tenant_1004/1/log", dir_fd:203, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:54585248, block_header_serialize_buf_:0x14b0efb300b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1713921656087129, this:0x14b0efb2c9f0}, [Reconfirm]={palf_id:1, self:"10.0.0.3:2882", state:"FINISHED", new_proposal_id:5, prepare_log_ack_list:1"10.0.0.2:2882";, curr_paxos_follower_list:1{server:"10.0.0.2:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"10.0.0.3:2882", majority_max_accept_pid:4, majority_max_lsn:{lsn:69192977}, saved_end_lsn:{lsn:69192977}, last_submit_prepare_req_time_us:1713921656087808, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:1713921656583907, last_purge_throttling_time_us:-1, this:0x14b0efb2c430}) [2024-04-24 10:00:48.691290] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2652) [874414][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=331] [PALF_DUMP](palf_id=1, self="10.0.0.3:2882", ack_info_list=[{member:{server:"10.0.0.2:2882", timestamp:1, flag:0}, last_ack_time_us:1713924048547729, last_flushed_end_lsn:{lsn:79617788}}, {member:{server:"10.0.0.3:2882", timestamp:1, flag:0}, last_ack_time_us:1713924048677182, last_flushed_end_lsn:{lsn:79624288}}]) [2024-04-24 10:00:48.691357] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4290) [874414][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=58] current log_task status(palf_id=1, self="10.0.0.3:2882", log_id=304857, log_task={header:{begin_lsn:{lsn:79617788}, end_lsn:{lsn:79618657}, log_id:304857, min_scn:{val:1713924048656929000, v:0}, max_scn:{val:1713924048656929000, v:0}, data_len:813, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:79617788}, data_checksum:3453388693, accum_checksum:2633745197, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1713924048657171, freeze_ts:1713924048657189, submit_ts:1713924048657312, flushed_ts:1713924048662930, gen_to_freeze cost time:18, gen_to_submit cost time:141, submit_to_flush cost time:5618}, ack_info_list=[{member:{server:"10.0.0.2:2882", timestamp:1, flag:0}, last_ack_time_us:1713924048547729, last_flushed_end_lsn:{lsn:79617788}}, {member:{server:"10.0.0.3:2882", timestamp:1, flag:0}, last_ack_time_us:1713924048677182, last_flushed_end_lsn:{lsn:79624288}}]) [2024-04-24 10:00:48.694805] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [874529][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=82] ====== check clog disk timer task ====== [2024-04-24 10:00:48.694920] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [874529][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=107] get_disk_usage(ret=0, capacity(MB):=33177, used(MB):=390) [2024-04-24 10:00:48.695046] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [874529][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=70] cannot_recycle_log_size statistics(cannot_recycle_log_size=208096441, threshold=10436770529, need_update_checkpoint_scn=false) [2024-04-24 10:00:48.703068] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=90] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.703666] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=116] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.703823] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=151] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.710932] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=77] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8905367552, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:48.714194] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1E97-0-0] [lt=135][errcode=0] set logical del time(cache_obj->get_logical_del_time()=576361288308, cache_obj->added_lc()=false, cache_obj->get_object_id()=39596, cache_obj->get_tenant_id()=1, lbt()="0x121f36ac 0xc327dd4 0x4e3fdbe 0x4e31e28 0x4e29b67 0x4e21201 0x4e1eb17 0x4e0e175 0xa7ac354 0x12d1fdbc 0x14b1c1d132de 0x14b1c1a44e83") [2024-04-24 10:00:48.718317] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1457) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2B1BC330E-0-0] [lt=184] receive register_req from acceptor, ignore(self="10.0.0.3:2882", palf_id=1, child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924048713682, keepalive_ts:-1}) [2024-04-24 10:00:48.718459] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1476) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2B1BC330E-0-0] [lt=130] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="10.0.0.3:2882", child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924048713682, keepalive_ts:-1}, is_to_leader=true) [2024-04-24 10:00:48.727697] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=75][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.729044] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=126] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.738155] INFO do_work (ob_rl_mgr.cpp:709) [861413][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=113] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-24 10:00:48.745157] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=101] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.750619] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [862276][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=88] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}, {id:1002}]) [2024-04-24 10:00:48.753344] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=90] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.754404] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B47F14AE-0-0] [lt=83] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="10.0.0.3:2882", committed size=3390) [2024-04-24 10:00:48.754617] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B47F14AE-0-0] [lt=130] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=28, this=0x14b13adc36a0, extra_info="palf_id:1002") [2024-04-24 10:00:48.754850] INFO [PALF] read (log_cache.cpp:100) [862236][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=69] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=3390, hit_cnt=4, read_cnt=4, hit rate=1.000000000000000000e+00) [2024-04-24 10:00:48.754963] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B47F14AE-0-0] [lt=321] [PALF STAT LOG LIFE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=430284, this=0x14b13adc38d0, extra_info="palf_id:1002") [2024-04-24 10:00:48.755044] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B47F14AE-0-0] [lt=74] [PALF STAT LOG TASK TIME](palf_id=1002, self="10.0.0.3:2882", total_slide_log_cnt=5, avg_log_gen_to_freeze_time=12, avg_log_gen_to_submit_time=91, avg_log_submit_to_flush_time=237509, avg_log_submit_to_slide_time=430193) [2024-04-24 10:00:48.755136] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B47F14AE-0-0] [lt=86][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=301035, log_task={header:{begin_lsn:{lsn:165100885}, end_lsn:{lsn:165102458}, log_id:301035, min_scn:{val:1713924048534839000, v:0}, max_scn:{val:1713924048534839000, v:0}, data_len:1517, proposal_id:5, prev_lsn:{lsn:165099312}, prev_proposal_id:5, committed_end_lsn:{lsn:165100885}, data_checksum:1911720022, accum_checksum:268244097, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924048537330, freeze_ts:1713924048537337, submit_ts:1713924048537394, flushed_ts:1713924048544763, gen_to_freeze cost time:7, gen_to_submit cost time:64, submit_to_flush cost time:7369}, fs_cb_begin_ts=1713924048754558, log_life_time=217228) [2024-04-24 10:00:48.755554] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [862286][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=84] current clock is (clock_=0, max_seq_=0, mem_limit=213070636, cur_mem_hold=109051904, attr_.tenant_id_=1002) [2024-04-24 10:00:48.756164] INFO [LIB] stat (utility.h:1197) [862286][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=140] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=214, this=0x14b16c6048f8, extra_info="") [2024-04-24 10:00:48.756286] INFO [LIB] stat (utility.h:1197) [862286][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=105] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=762, this=0x14b16c604b28, extra_info="") [2024-04-24 10:00:48.759366] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [882037][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E99-0-0] [lt=68][errcode=-4023] tx is 2pc logging(this={this:0x14b168f2d710, ref:3, trans_id:{txid:877463}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924648560497, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048574692], ctx_create_time:1713924048560497}{ls_id:{id:1001}, session_id:3221998686, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048572105, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048573312000, v:0}, log_ts:{val:1713924048573312000, v:0}, lsn:{lsn:1002193683}, submit_ts:1713924048573312}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877463}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877463}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048573312000, v:0}, start_scn:{val:1713924048573312000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924048573312000, v:0} commit_version={val:1713924048573312000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877463} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=2617039338 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048573312000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048573312, block_frozen_memtable:null}) [2024-04-24 10:00:48.759617] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [882037][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E99-0-0] [lt=233][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b168f2d710, ref:3, trans_id:{txid:877463}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924648560497, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048574692], ctx_create_time:1713924048560497}{ls_id:{id:1001}, session_id:3221998686, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048572105, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048573312000, v:0}, log_ts:{val:1713924048573312000, v:0}, lsn:{lsn:1002193683}, submit_ts:1713924048573312}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877463}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877463}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048573312000, v:0}, start_scn:{val:1713924048573312000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924048573312000, v:0} commit_version={val:1713924048573312000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877463} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=2617039338 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048573312000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048573312, block_frozen_memtable:null}) [2024-04-24 10:00:48.759804] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [882037][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E99-0-0] [lt=169][errcode=-4023] commit fail(ret=-4023, coord={id:1001}, tx_id={txid:877463}) [2024-04-24 10:00:48.759955] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [882037][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E99-0-0] [lt=134][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877463}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924048754708, cluster_id:1713874979}, expire_ts:1713924618349599, commit_start_scn:{val:1713924048554997000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:48.760133] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [882037][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E99-0-0] [lt=156][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877463}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924048754708, cluster_id:1713874979}, expire_ts:1713924618349599, commit_start_scn:{val:1713924048554997000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:48.762394] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=95][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.774064] INFO [LIB] stat (utility.h:1197) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=74] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=93, this=0x14b0efb29470, extra_info="") [2024-04-24 10:00:48.774204] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=106] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=38878, avg_group_log_size=3887) [2024-04-24 10:00:48.774359] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=100] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=38756) [2024-04-24 10:00:48.774563] INFO [LIB] stat (utility.h:1197) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=166] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x14b0efafce50, extra_info="id:1") [2024-04-24 10:00:48.774694] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=124] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.774998] INFO [LIB] stat (utility.h:1197) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=90] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=95, this=0x14b0c0e2b470, extra_info="") [2024-04-24 10:00:48.775139] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=101] [PALF STAT GROUP LOG INFO](palf_id=1001, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=6828, avg_group_log_size=569) [2024-04-24 10:00:48.775268] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=91] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.775388] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=111] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.779727] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [862534][T1002_L0_G5][T1002][YB420A000002-000616C2BEDDD119-0-0] [lt=69] get number(ret=0, service_type_=0, range=1, base_id=1713924048779042000, start_id=1713924048779042000, end_id=1713924048779042001) [2024-04-24 10:00:48.779769] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [862476][T1001_L0_G5][T1001][YB420A000002-000616C2BEDDD118-0-0] [lt=152] get number(ret=0, service_type_=0, range=1, base_id=1713924048779042000, start_id=1713924048779042000, end_id=1713924048779042001) [2024-04-24 10:00:48.783549] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=130] new mysql sessid created(conn.sessid_=3221542293, support_ssl=false) [2024-04-24 10:00:48.783768] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=150] sm conn init succ(conn.sessid_=3221542293, sess.client_addr_="10.0.0.3:48124") [2024-04-24 10:00:48.783970] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=186] accept one succ(*s={this:0x14b19e892110, session_id:3221542293, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:168}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:48.784219] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:943) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=154][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x14b19e892110, session_id:3221542293, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:168}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:48.784303] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=76] kill and revert session(conn.sessid_=3221542293, proxy_sessid=0, server_id=1, ret=0) [2024-04-24 10:00:48.784392] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:918) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=78] sock ref clean, do destroy(*s={this:0x14b19e892110, session_id:3221542293, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:168}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:48.784525] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:223) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=123] mark session id unused(conn.sessid_=3221542293) [2024-04-24 10:00:48.784597] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [861519][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=59] connection close(sessid=3221542293, proxy_sessid=0, tenant_id=0, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:0], disconnect_state=0) [2024-04-24 10:00:48.803083] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=134] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.803221] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=124] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.803345] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=111] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.805480] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB8FE-0-0] [lt=76] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=5565) [2024-04-24 10:00:48.805679] INFO [LIB] stat (utility.h:1197) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB8FE-0-0] [lt=120] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=29, this=0x14b14c9b36a0, extra_info="palf_id:1") [2024-04-24 10:00:48.805840] INFO [LIB] stat (utility.h:1197) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB8FE-0-0] [lt=134] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=443193, this=0x14b14c9b38d0, extra_info="palf_id:1") [2024-04-24 10:00:48.805813] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=74][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048563114000, v:0}, log_ts:{val:1713924048563114000, v:0}, lsn:{lsn:314132013}, submit_ts:1713924048563114}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:568571}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=242676, ctx_lock_wait_time=10) [2024-04-24 10:00:48.806009] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB8FE-0-0] [lt=134] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=7, avg_log_gen_to_freeze_time=158929, avg_log_gen_to_submit_time=159109, avg_log_submit_to_flush_time=9729, avg_log_submit_to_slide_time=284084) [2024-04-24 10:00:48.806127] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB8FE-0-0] [lt=109][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=505461, log_task={header:{begin_lsn:{lsn:314131957}, end_lsn:{lsn:314132499}, log_id:505461, min_scn:{val:1713924048563114000, v:0}, max_scn:{val:1713924048563114000, v:0}, data_len:486, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:314131957}, data_checksum:3378598597, accum_checksum:1208263920, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924048563817, freeze_ts:1713924048563827, submit_ts:1713924048564030, flushed_ts:1713924048573123, gen_to_freeze cost time:10, gen_to_submit cost time:213, submit_to_flush cost time:9093}, fs_cb_begin_ts=1713924048805611, log_life_time=241794) [2024-04-24 10:00:48.806445] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [862260][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=95] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1713924048246903000, v:0}, new_max_version={val:1713924048246903000, v:0}, last_min_version={val:1713924047777372000, v:0}, last_max_version={val:1713924047777372000, v:0}, total_time_us=244836, delta_us=559532) [2024-04-24 10:00:48.806552] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [862260][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=100] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1713924048246903000, v:0}, min_version={val:1713924048246903000, v:0}, max_version={val:1713924048246903000, v:0}, in_service=true, leader_epoch=5) [2024-04-24 10:00:48.809699] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD28D-0-0] [lt=111] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="10.0.0.3:2882", committed size=7722) [2024-04-24 10:00:48.809959] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD28D-0-0] [lt=129] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=31, this=0x14b0c0e2b6a0, extra_info="palf_id:1001") [2024-04-24 10:00:48.810113] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD28D-0-0] [lt=132] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=331076, this=0x14b0c0e2b8d0, extra_info="palf_id:1001") [2024-04-24 10:00:48.810113] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2C0A8F1B4-0-0] [lt=119][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048588043000, v:0}, log_ts:{val:1713924048588043000, v:0}, lsn:{lsn:88120220}, submit_ts:1713924048588043}, this:0x14b1481b8b40, is_inited_:true, trans_id:{txid:327196}, ls_id:{id:1001}, ctx:0x14b1481b5bd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=222048, ctx_lock_wait_time=11) [2024-04-24 10:00:48.810269] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD28D-0-0] [lt=145] [PALF STAT LOG TASK TIME](palf_id=1001, self="10.0.0.3:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=67475, avg_log_gen_to_submit_time=67631, avg_log_submit_to_flush_time=11870, avg_log_submit_to_slide_time=263445) [2024-04-24 10:00:48.810422] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD28D-0-0] [lt=138][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=330728, log_task={header:{begin_lsn:{lsn:88120164}, end_lsn:{lsn:88120834}, log_id:330728, min_scn:{val:1713924048588043000, v:0}, max_scn:{val:1713924048588043000, v:0}, data_len:614, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:88120164}, data_checksum:4077380452, accum_checksum:4292651242, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924048588608, freeze_ts:1713924048588617, submit_ts:1713924048588763, flushed_ts:1713924048596621, gen_to_freeze cost time:9, gen_to_submit cost time:155, submit_to_flush cost time:7858}, fs_cb_begin_ts=1713924048809849, log_life_time=221241) [2024-04-24 10:00:48.815961] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0530-0-0] [lt=98][errcode=-4023] tx is 2pc logging(this={this:0x14b1481c0010, ref:5, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048664959], ctx_create_time:1713924045155303}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79589464}, {lsn:79618713}], redo_log_no:2, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048664122000, v:0}, trans_type:2, next_log_entry_no:3, max_applied_log_ts:{val:1713924048272353001, v:0}, max_applying_log_ts:{val:1713924048272353001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:1713924048659397, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:79589464}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924048272353001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877443458344 max_table_version=1713877467338312 trans_version={val:1713924048664122000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1 callback_alloc_count=134 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=158 redo_sync_succ_count=130 redo_sync_fail_count=0 main_list_length=158 unsynced_cnt=21 unsubmitted_cnt_=0 cb_statistics:[main=158, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924048272353001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048665345, block_frozen_memtable:null}) [2024-04-24 10:00:48.816288] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0530-0-0] [lt=316][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b1481c0010, ref:5, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048664959], ctx_create_time:1713924045155303}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79589464}, {lsn:79618713}], redo_log_no:2, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048664122000, v:0}, trans_type:2, next_log_entry_no:3, max_applied_log_ts:{val:1713924048272353001, v:0}, max_applying_log_ts:{val:1713924048272353001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:1713924048659397, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:79589464}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924048272353001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877443458344 max_table_version=1713877467338312 trans_version={val:1713924048664122000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1 callback_alloc_count=134 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=158 redo_sync_succ_count=130 redo_sync_fail_count=0 main_list_length=158 unsynced_cnt=21 unsubmitted_cnt_=0 cb_statistics:[main=158, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924048272353001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048665345, block_frozen_memtable:null}) [2024-04-24 10:00:48.816553] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0530-0-0] [lt=245][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176888}) [2024-04-24 10:00:48.816751] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0530-0-0] [lt=137][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176888}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:37, timestamp:1713924048810562, cluster_id:1713874979}, expire_ts:1713925045138383, commit_start_scn:{val:1713924048658150000, v:0}, parts:[{id:1}, {id:1001}], app_trace_info:""}) [2024-04-24 10:00:48.816947] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0530-0-0] [lt=170][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176888}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:37, timestamp:1713924048810562, cluster_id:1713874979}, expire_ts:1713925045138383, commit_start_scn:{val:1713924048658150000, v:0}, parts:[{id:1}, {id:1001}], app_trace_info:""}) [2024-04-24 10:00:48.819103] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C44-0-0] [lt=85] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="10.0.0.3:2882", committed size=9671) [2024-04-24 10:00:48.819186] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F76-0-0] [lt=204] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=7075) [2024-04-24 10:00:48.819284] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C44-0-0] [lt=94] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=27, this=0x14b1336eb6a0, extra_info="palf_id:1001") [2024-04-24 10:00:48.819326] INFO [LIB] stat (utility.h:1197) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F76-0-0] [lt=80] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=28, this=0x14b0f0aa16a0, extra_info="palf_id:1") [2024-04-24 10:00:48.819377] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=74][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048573312000, v:0}, log_ts:{val:1713924048573312000, v:0}, lsn:{lsn:1002193683}, submit_ts:1713924048573312}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877463}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=246039, ctx_lock_wait_time=14) [2024-04-24 10:00:48.819412] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C44-0-0] [lt=106] [PALF STAT LOG LIFE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=482388, this=0x14b1336eb8d0, extra_info="palf_id:1001") [2024-04-24 10:00:48.819439] INFO [LIB] stat (utility.h:1197) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F76-0-0] [lt=95] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=377994, this=0x14b0f0aa18d0, extra_info="palf_id:1") [2024-04-24 10:00:48.819455] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=110][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048596957000, v:0}, log_ts:{val:1713924048596957000, v:0}, lsn:{lsn:298519494}, submit_ts:1713924048596957}, this:0x14b0fbfc3d40, is_inited_:true, trans_id:{txid:548729}, ls_id:{id:1}, ctx:0x14b0fbfc0dd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=222482, ctx_lock_wait_time=7) [2024-04-24 10:00:48.819484] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C44-0-0] [lt=66] [PALF STAT LOG TASK TIME](palf_id=1001, self="10.0.0.3:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=141855, avg_log_gen_to_submit_time=142029, avg_log_submit_to_flush_time=9809, avg_log_submit_to_slide_time=340358) [2024-04-24 10:00:48.819534] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F76-0-0] [lt=70] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=106801, avg_log_gen_to_submit_time=106937, avg_log_submit_to_flush_time=9339, avg_log_submit_to_slide_time=271056) [2024-04-24 10:00:48.819597] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C44-0-0] [lt=103][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=673421, log_task={header:{begin_lsn:{lsn:1002193627}, end_lsn:{lsn:1002195200}, log_id:673421, min_scn:{val:1713924048573312000, v:0}, max_scn:{val:1713924048573312000, v:0}, data_len:1517, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1002193627}, data_checksum:2720297693, accum_checksum:1173749694, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924048574772, freeze_ts:1713924048574782, submit_ts:1713924048574959, flushed_ts:1713924048586387, gen_to_freeze cost time:10, gen_to_submit cost time:187, submit_to_flush cost time:11428}, fs_cb_begin_ts=1713924048819211, log_life_time=244439) [2024-04-24 10:00:48.819684] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F76-0-0] [lt=119][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=482552, log_task={header:{begin_lsn:{lsn:298519438}, end_lsn:{lsn:298520411}, log_id:482552, min_scn:{val:1713924048596957000, v:0}, max_scn:{val:1713924048596957000, v:0}, data_len:917, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:298519438}, data_checksum:996662366, accum_checksum:2347386729, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924048597307, freeze_ts:1713924048597320, submit_ts:1713924048597484, flushed_ts:1713924048616318, gen_to_freeze cost time:13, gen_to_submit cost time:177, submit_to_flush cost time:18834}, fs_cb_begin_ts=1713924048819276, log_life_time=221969) [2024-04-24 10:00:48.819726] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=101] rpc statistics(total_trans_msg_count=11, total_batch_msg_count=0) [2024-04-24 10:00:48.820081] INFO [LIB] stat (utility.h:1197) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141B-0-0] [lt=101] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=10, this=0x14b186b47470, extra_info="") [2024-04-24 10:00:48.820100] INFO [SERVER] do_batch_update_ (ob_tablet_table_updater.cpp:645) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F3-0-0] [lt=92] REPORT: batch update tablets finished(ret=0, ret="OB_SUCCESS", replicas.count()=1, cost_time=227905) [2024-04-24 10:00:48.820231] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141B-0-0] [lt=109] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="FOLLOWER", total_group_log_cnt=7, avg_log_batch_cnt=0, total_group_log_size=5911, avg_group_log_size=844) [2024-04-24 10:00:48.820359] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141B-0-0] [lt=118] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=6335) [2024-04-24 10:00:48.820618] INFO [LIB] stat (utility.h:1197) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141B-0-0] [lt=89] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=32, this=0x14b186b476a0, extra_info="palf_id:1") [2024-04-24 10:00:48.820813] INFO [LIB] stat (utility.h:1197) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141B-0-0] [lt=135] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=324325, this=0x14b186b478d0, extra_info="palf_id:1") [2024-04-24 10:00:48.820828] INFO [PALF] read (log_cache.cpp:100) [861710][T1_ReplaySrv2][T1][Y0-0000000000000000-0-0] [lt=6638] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=6335, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2024-04-24 10:00:48.820828] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD28E-0-0] [lt=183] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=37823) [2024-04-24 10:00:48.820962] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141B-0-0] [lt=136] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=12, avg_log_gen_to_submit_time=117, avg_log_submit_to_flush_time=214318, avg_log_submit_to_slide_time=324208) [2024-04-24 10:00:48.821040] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD28E-0-0] [lt=149] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=30, this=0x14b0efb296a0, extra_info="palf_id:1") [2024-04-24 10:00:48.821059] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141B-0-0] [lt=78][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=364241, log_task={header:{begin_lsn:{lsn:179744364}, end_lsn:{lsn:179745436}, log_id:364241, min_scn:{val:1713924048594493000, v:0}, max_scn:{val:1713924048594493000, v:0}, data_len:1016, proposal_id:5, prev_lsn:{lsn:179743336}, prev_proposal_id:5, committed_end_lsn:{lsn:179744364}, data_checksum:177810747, accum_checksum:1038924345, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924048600752, freeze_ts:1713924048600761, submit_ts:1713924048600823, flushed_ts:1713924048618365, gen_to_freeze cost time:9, gen_to_submit cost time:71, submit_to_flush cost time:17542}, fs_cb_begin_ts=1713924048820471, log_life_time=219719) [2024-04-24 10:00:48.821150] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD28E-0-0] [lt=95] [PALF STAT LOG LIFE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=391220, this=0x14b0efb298d0, extra_info="palf_id:1") [2024-04-24 10:00:48.821193] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2A1-0-0] [lt=115][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048656929000, v:0}, log_ts:{val:1713924048656929000, v:0}, lsn:{lsn:79617844}, submit_ts:1713924048656929}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176912}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=164242, ctx_lock_wait_time=7) [2024-04-24 10:00:48.821229] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD28E-0-0] [lt=73] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=146456, avg_log_gen_to_submit_time=146632, avg_log_submit_to_flush_time=7478, avg_log_submit_to_slide_time=244588) [2024-04-24 10:00:48.821296] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD28E-0-0] [lt=60][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=304857, log_task={header:{begin_lsn:{lsn:79617788}, end_lsn:{lsn:79618657}, log_id:304857, min_scn:{val:1713924048656929000, v:0}, max_scn:{val:1713924048656929000, v:0}, data_len:813, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:79617788}, data_checksum:3453388693, accum_checksum:2633745197, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924048657171, freeze_ts:1713924048657189, submit_ts:1713924048657312, flushed_ts:1713924048662930, gen_to_freeze cost time:18, gen_to_submit cost time:141, submit_to_flush cost time:5618}, fs_cb_begin_ts=1713924048820994, log_life_time=163823) [2024-04-24 10:00:48.821437] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2A1-0-0] [lt=90] rpc statistics(total_trans_msg_count=5, total_batch_msg_count=1) [2024-04-24 10:00:48.823072] INFO [LIB] stat (utility.h:1197) [861715][T1_ReplaySrv7][T1][Y0-0000000000000000-0-0] [lt=75] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=375, this=0x14b1976048f8, extra_info="") [2024-04-24 10:00:48.823187] INFO [LIB] stat (utility.h:1197) [861715][T1_ReplaySrv7][T1][Y0-0000000000000000-0-0] [lt=91] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1125, this=0x14b197604b28, extra_info="") [2024-04-24 10:00:48.825064] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [861423][TbltTblUp0][T1003][YB420A000003-000616C2AB2901F4-0-0] [lt=131] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=6491) [2024-04-24 10:00:48.825190] INFO [LIB] stat (utility.h:1197) [861423][TbltTblUp0][T1003][YB420A000003-000616C2AB2901F4-0-0] [lt=91] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x14b0f0a74e50, extra_info="id:1") [2024-04-24 10:00:48.826461] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:860) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=94] memtable handle rollback to successfuly(from_seq_no=1713924048825484, to_seq_no=1713924048820305, *this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877464} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=1, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}) [2024-04-24 10:00:48.826745] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=196][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048641803000, v:0}, log_ts:{val:1713924048641803000, v:0}, lsn:{lsn:314132555}, submit_ts:1713924048641803}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568572}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=184919, ctx_lock_wait_time=14) [2024-04-24 10:00:48.827943] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=73][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.829342] INFO [LIB] stat (utility.h:1197) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=86] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=101, this=0x14b14c9b3470, extra_info="") [2024-04-24 10:00:48.829482] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=92] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=8, avg_log_batch_cnt=1, total_group_log_size=4370, avg_group_log_size=546) [2024-04-24 10:00:48.829626] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=112] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.830407] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2A1-0-0] [lt=121][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048658150001, v:0}, log_ts:{val:1713924048658150001, v:0}, lsn:{lsn:88121631}, submit_ts:1713924048666619}, this:0x14b055e20af0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x40, arg:null}, {log_type:0x80, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=163772, ctx_lock_wait_time=10) [2024-04-24 10:00:48.830560] INFO [MDS.EVENT]try_decline_rec_scn (mds_table_base.cpp:250) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2A1-0-0] [lt=85] DECLINE_REC_SCN(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:200317}}, event={alloc:null, timestamp:"2024-04-24 10:00:48.830552", event:"DECLINE_REC_SCN", info_str:"{val:4611686018427387903, v:0} -> {val:1713924048588043001, v:0}", unit_id:255, key_str:"", writer_type:0, writer_id:0, seq_no:0, redo_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, trans_version:{val:18446744073709551615, v:3}, node_type:0, state:5}) [2024-04-24 10:00:48.830743] INFO [MDS.EVENT]on_redo_ (mds_node.ipp:150) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2A1-0-0] [lt=138] ON_REDO(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:200317}}, event={alloc:null, timestamp:"2024-04-24 10:00:48.830728", event:"ON_REDO", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176888, seq_no:0, redo_scn:{val:1713924048588043001, v:0}, end_scn:{val:4611686018427387903, v:0}, trans_version:{val:0, v:0}, node_type:1, state:1}) [2024-04-24 10:00:48.830936] INFO [MDS.EVENT]try_decline_rec_scn (mds_table_base.cpp:250) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2A1-0-0] [lt=95] DECLINE_REC_SCN(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:1152921504606847581}}, event={alloc:null, timestamp:"2024-04-24 10:00:48.830923", event:"DECLINE_REC_SCN", info_str:"{val:4611686018427387903, v:0} -> {val:1713924048588043001, v:0}", unit_id:255, key_str:"", writer_type:0, writer_id:0, seq_no:0, redo_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, trans_version:{val:18446744073709551615, v:3}, node_type:0, state:5}) [2024-04-24 10:00:48.831103] INFO [MDS.EVENT]on_redo_ (mds_node.ipp:150) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2A1-0-0] [lt=127] ON_REDO(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:1152921504606847581}}, event={alloc:null, timestamp:"2024-04-24 10:00:48.831087", event:"ON_REDO", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176888, seq_no:0, redo_scn:{val:1713924048588043001, v:0}, end_scn:{val:4611686018427387903, v:0}, trans_version:{val:0, v:0}, node_type:1, state:1}) [2024-04-24 10:00:48.831224] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2A1-0-0] [lt=98] notify MDS(ret=0, trans_id_={txid:176888}, ls_id_={id:1001}, notify_type=1, log_ts={val:1713924048588043001, v:0}, notify_array.count()=1, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], total_time=0) [2024-04-24 10:00:48.831386] INFO [MDS.EVENT]on_prepare_ (mds_node.ipp:202) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2A1-0-0] [lt=103] ON_PREPARE(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:200317}}, event={alloc:null, timestamp:"2024-04-24 10:00:48.831369", event:"ON_PREPARE", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176888, seq_no:0, redo_scn:{val:1713924048588043001, v:0}, end_scn:{val:4611686018427387903, v:0}, trans_version:{val:0, v:0}, node_type:1, state:2}) [2024-04-24 10:00:48.831550] INFO [MDS.EVENT]on_prepare_ (mds_node.ipp:202) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2A1-0-0] [lt=102] ON_PREPARE(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:1152921504606847581}}, event={alloc:null, timestamp:"2024-04-24 10:00:48.831528", event:"ON_PREPARE", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176888, seq_no:0, redo_scn:{val:1713924048588043001, v:0}, end_scn:{val:4611686018427387903, v:0}, trans_version:{val:0, v:0}, node_type:1, state:2}) [2024-04-24 10:00:48.831608] WDIAG [STORAGE.TRANS] inner_lock_for_read (ob_tx_data_functor.cpp:290) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2AEAC0CD0-0-0] [lt=90][errcode=-6004] lock_for_read need retry(ret=-6004, tx_data={tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924048272353001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}, lock_for_read_arg_={mvcc_acc_ctx_:{type:1, abs_lock_timeout:1713924148726364, tx_lock_timeout:-1, snapshot:{version:{val:1713924048823139000, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x14b0efafe490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, data_trans_id_:{txid:176888}, data_sql_sequence_:1713924048475558, read_latest_:false, scn_:{val:4611686018427387903, v:0}}, tx_cc_ctx={state:10, prepare_version:{val:1713924048664122000, v:0}}) [2024-04-24 10:00:48.831661] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2A1-0-0] [lt=75] notify MDS(ret=0, trans_id_={txid:176888}, ls_id_={id:1001}, notify_type=3, log_ts={val:1713924048658150001, v:0}, notify_array.count()=1, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], total_time=0) [2024-04-24 10:00:48.831952] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=85][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:0, v:0}, log_ts:{val:1713924048588043001, v:0}, lsn:{lsn:88120890}, submit_ts:1713924048665345}, this:0x14b055e208a8, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:true, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x4, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=165046, ctx_lock_wait_time=1553) [2024-04-24 10:00:48.835840] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=141] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-04-24 10:00:48.836429] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=104] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, append size=8220) [2024-04-24 10:00:48.836558] INFO [LIB] stat (utility.h:1197) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=106] [PALF STAT APPEND COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x14b134390e50, extra_info="id:1001") [2024-04-24 10:00:48.837090] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [861692][T1_LSArchiveMgr][T1][YB420A000003-000616C2A888F0E7-0-0] [lt=131] gc stale ls task succ [2024-04-24 10:00:48.839074] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874554][T1004_ApplySrv2][T1004][YB420A000003-000616C2BFC8F285-0-0] [lt=137][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:0, v:0}, log_ts:{val:1713924048656929001, v:0}, lsn:{lsn:79618713}, submit_ts:1713924048664122}, this:0x14b1481c2868, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():1, range_array_:[{register_no_:1, has_submitted_:true, has_synced_:false, type:6, data_.length():1313}]}, cb_arg_array_:[{log_type:0x4, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=174933, ctx_lock_wait_time=12) [2024-04-24 10:00:48.839205] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874554][T1004_ApplySrv2][T1004][YB420A000003-000616C2BFC8F285-0-0] [lt=112] notify MDS(ret=0, trans_id_={txid:176888}, ls_id_={id:1}, notify_type=1, log_ts={val:1713924048656929001, v:0}, notify_array.count()=1, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:true, type:6, data_.length():1313}], total_time=0) [2024-04-24 10:00:48.839336] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=95][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048664122000, v:0}, lsn:{lsn:79620195}, submit_ts:1713924048665345}, this:0x14b1481c2ab0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x80, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=173860, ctx_lock_wait_time=121) [2024-04-24 10:00:48.839459] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=95] notify MDS(ret=0, trans_id_={txid:176888}, ls_id_={id:1}, notify_type=3, log_ts={val:1713924048664122000, v:0}, notify_array.count()=1, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:true, type:6, data_.length():1313}], total_time=0) [2024-04-24 10:00:48.842772] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [874542][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=93] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}, {id:1002}]) [2024-04-24 10:00:48.842936] INFO [CLOG] run1 (ob_garbage_collector.cpp:1338) [861667][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=71] Garbage Collector is running(seq_=4886, gc_interval=10000000) [2024-04-24 10:00:48.845392] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=82] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.847025] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:272) [861667][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=74] GC check ls in member list, skip it(id={id:1}, leader="10.0.0.2:2882") [2024-04-24 10:00:48.847148] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1431) [861667][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=101] gc_check_member_list_ cost time(ret=0, time_us=4120) [2024-04-24 10:00:48.847258] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [861667][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=77] execute_gc cost time(ret=0, time_us=13) [2024-04-24 10:00:48.847415] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1626) [861667][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=93] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2024-04-24 10:00:48.847567] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1648) [861667][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=102] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2024-04-24 10:00:48.847707] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [861667][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=100] execute_gc cost time(ret=0, time_us=171) [2024-04-24 10:00:48.847800] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [861667][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=78] ObSafeDestroyHandler start process [2024-04-24 10:00:48.853511] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=116] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.862549] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=103][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.871795] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=108][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67159217}], scn_array=[{val:1713924048573312000, v:0}], curr_size=122, accum_size=366, time_cost=168580) [2024-04-24 10:00:48.872264] INFO [LIB] stat (utility.h:1197) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=126] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=26926, this=0x14b1336f53b0, extra_info="palf_id:1001") [2024-04-24 10:00:48.872361] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=88][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:1002195200}], scn_array=[{val:1713924048573312001, v:0}], curr_size=122, accum_size=9793, time_cost=168415) [2024-04-24 10:00:48.872361] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=157] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=38, this=0x14b1470ab6a0, extra_info="palf_id:1") [2024-04-24 10:00:48.872466] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=91] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=9793) [2024-04-24 10:00:48.872548] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=105] [PALF STAT LOG LIFE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=151766, this=0x14b1470ab8d0, extra_info="palf_id:1") [2024-04-24 10:00:48.872608] INFO [LIB] stat (utility.h:1214) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=102] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=355, this=0x14b1679f6c70, extra_info="") [2024-04-24 10:00:48.872724] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=142] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=5, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=212, avg_log_submit_to_flush_time=56004, avg_log_submit_to_slide_time=151553) [2024-04-24 10:00:48.872925] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=195][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=259317, log_task={header:{begin_lsn:{lsn:67159217}, end_lsn:{lsn:67159339}, log_id:259317, min_scn:{val:1713924048573312000, v:0}, max_scn:{val:1713924048573312000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:67159217}, data_checksum:667309872, accum_checksum:504190444, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924048702731, freeze_ts:1713924048702743, submit_ts:1713924048702996, flushed_ts:1713924048872228, gen_to_freeze cost time:12, gen_to_submit cost time:265, submit_to_flush cost time:169232}, fs_cb_begin_ts=1713924048872295, log_life_time=169564) [2024-04-24 10:00:48.873227] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=87] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=101, this=0x14b1336eb470, extra_info="") [2024-04-24 10:00:48.873397] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=90] [PALF STAT GROUP LOG INFO](palf_id=1001, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=9793, avg_group_log_size=1088) [2024-04-24 10:00:48.874167] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=104] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=182, this=0x14b1336f55e0, extra_info="palf_id:1001") [2024-04-24 10:00:48.874201] INFO [LIB] stat (utility.h:1197) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=214] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=21665, this=0x14b0f0aab3b0, extra_info="palf_id:1") [2024-04-24 10:00:48.874290] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=82][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:298520411}], scn_array=[{val:1713924048721822000, v:0}], curr_size=542, accum_size=7617, time_cost=150792) [2024-04-24 10:00:48.874362] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=64] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=7617) [2024-04-24 10:00:48.874517] INFO [LIB] stat (utility.h:1214) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=89] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=179, this=0x14b1421a2530, extra_info="") [2024-04-24 10:00:48.874942] INFO [LIB] stat (utility.h:1197) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=58] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=97, this=0x14b0f0aa1470, extra_info="") [2024-04-24 10:00:48.875069] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=91] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=7381, avg_group_log_size=671) [2024-04-24 10:00:48.875256] INFO [LIB] stat (utility.h:1197) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=89] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=159, this=0x14b0f0aab5e0, extra_info="palf_id:1") [2024-04-24 10:00:48.875517] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=111][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048721822000, v:0}, log_ts:{val:1713924048721822000, v:0}, lsn:{lsn:298520467}, submit_ts:1713924048721822}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:548730}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=153676, ctx_lock_wait_time=11) [2024-04-24 10:00:48.876016] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=141] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.876166] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=132] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.877964] INFO [LIB] stat (utility.h:1197) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=159] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=12482, this=0x14b14c9bd3b0, extra_info="palf_id:1") [2024-04-24 10:00:48.878046] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=72][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:314133041}], scn_array=[{val:1713924048641803001, v:0}], curr_size=122, accum_size=6229, time_cost=48337) [2024-04-24 10:00:48.878134] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=74] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=6229) [2024-04-24 10:00:48.878243] INFO [LIB] stat (utility.h:1214) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=68] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=658, this=0x14b157ff6530, extra_info="") [2024-04-24 10:00:48.878459] INFO [LIB] stat (utility.h:1197) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=71] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=61, this=0x14b14c9bd5e0, extra_info="palf_id:1") [2024-04-24 10:00:48.880511] INFO [LIB] stat (utility.h:1197) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=184] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=28130, this=0x14b13adcd3b0, extra_info="palf_id:1002") [2024-04-24 10:00:48.880663] INFO [LIB] stat (utility.h:1197) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=112] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=16003, this=0x14b186b513b0, extra_info="palf_id:1") [2024-04-24 10:00:48.880744] INFO [LIB] stat (utility.h:1197) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=94] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=14765, this=0x14b0efb333b0, extra_info="palf_id:1") [2024-04-24 10:00:48.880753] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=222][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:165102458}], scn_array=[{val:1713924048554997000, v:0}], curr_size=122, accum_size=5085, time_cost=125915) [2024-04-24 10:00:48.880744] INFO [LIB] stat (utility.h:1197) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=106] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=18162, this=0x14b0c0e353b0, extra_info="palf_id:1001") [2024-04-24 10:00:48.880820] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=146][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179745436}], scn_array=[{val:1713924048656129000, v:0}], curr_size=604, accum_size=8763, time_cost=60169) [2024-04-24 10:00:48.881024] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=267][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:79624288}], scn_array=[{val:1713924048664122001, v:0}], curr_size=122, accum_size=43576, time_cost=106419) [2024-04-24 10:00:48.881034] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=280][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:88121786}], scn_array=[{val:1713924048664122000, v:0}], curr_size=122, accum_size=8796, time_cost=105437) [2024-04-24 10:00:48.881024] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=258] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=5085) [2024-04-24 10:00:48.881061] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=234] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=8763) [2024-04-24 10:00:48.881128] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=87] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=8796) [2024-04-24 10:00:48.881121] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=88] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=43576) [2024-04-24 10:00:48.881170] INFO [LIB] stat (utility.h:1214) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=97] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=190459, this=0x14b1679f73b0, extra_info="") [2024-04-24 10:00:48.881199] INFO [LIB] stat (utility.h:1214) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=82] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=183122, this=0x14b1945fe530, extra_info="") [2024-04-24 10:00:48.881254] INFO [LIB] stat (utility.h:1214) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=81] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=159, this=0x14b176ffcc70, extra_info="") [2024-04-24 10:00:48.881261] INFO [LIB] stat (utility.h:1214) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=96] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=380, this=0x14b176ffc530, extra_info="") [2024-04-24 10:00:48.881308] INFO [LIB] stat (utility.h:1197) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=82] [PALF STAT LOG IO WORKER QUEUE SIZE](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=2, this=0x14b1945fe1e8, extra_info="log_io_worker_num:1, cb_thread_pool_tg_id:297") [2024-04-24 10:00:48.881406] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=144] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=113, this=0x14b13adcd5e0, extra_info="palf_id:1002") [2024-04-24 10:00:48.881420] INFO [LIB] stat (utility.h:1197) [861568][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=104] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=120, this=0x14b186b515e0, extra_info="palf_id:1") [2024-04-24 10:00:48.881912] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=91] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=157, this=0x14b0c0e355e0, extra_info="palf_id:1001") [2024-04-24 10:00:48.882550] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=126] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=173, this=0x14b0efb335e0, extra_info="palf_id:1") [2024-04-24 10:00:48.882544] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2169) [862421][T1001_LSMetaCh][T1][YB420A000003-000616C2AF79AD74-0-0] [lt=61] need check schema(new_schema_version=1713878795542168, cached_tenant_schema_version=1713878795542168, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT * FROM __all_ls_meta_table WHERE tenant_id = ? ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-24 10:00:48.885707] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000003-000616C2B35ECB51-0-0] [lt=155][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=301036, log_task={header:{begin_lsn:{lsn:165102458}, end_lsn:{lsn:165102580}, log_id:301036, min_scn:{val:1713924048554997000, v:0}, max_scn:{val:1713924048554997000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:165100885}, prev_proposal_id:5, committed_end_lsn:{lsn:165102458}, data_checksum:1715074492, accum_checksum:144396744, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924048754273, freeze_ts:1713924048754281, submit_ts:1713924048754322, flushed_ts:1713924048881281, gen_to_freeze cost time:8, gen_to_submit cost time:49, submit_to_flush cost time:126959}, fs_cb_begin_ts=1713924048885604, log_life_time=131331) [2024-04-24 10:00:48.891979] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2C20C019D-0-0] [lt=109] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=8, this=0x14b13adc3470, extra_info="") [2024-04-24 10:00:48.892165] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2C20C019D-0-0] [lt=143] [PALF STAT GROUP LOG INFO](palf_id=1002, self="10.0.0.3:2882", role="FOLLOWER", total_group_log_cnt=5, avg_log_batch_cnt=0, total_group_log_size=4963, avg_group_log_size=992) [2024-04-24 10:00:48.892184] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=98][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179746040}, {lsn:179747121}], scn_array=[{val:1713924048819761000, v:0}, {val:1713924048840067000, v:0}], curr_size=2074, accum_size=2074, time_cost=10675) [2024-04-24 10:00:48.892704] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=108][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:298520953}], scn_array=[{val:1713924048725291000, v:0}], curr_size=1376, accum_size=1376, time_cost=17487) [2024-04-24 10:00:48.894241] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=82][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:88121908}], scn_array=[{val:1713924048823139000, v:0}], curr_size=1702, accum_size=1702, time_cost=12290) [2024-04-24 10:00:48.894467] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=86][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:79624410}], scn_array=[{val:1713924048839739000, v:0}], curr_size=2240, accum_size=2240, time_cost=11716) [2024-04-24 10:00:48.895304] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [861710][T1_ReplaySrv2][T1][Y0-0000000000000000-0-0] [lt=134] current clock is (clock_=0, max_seq_=0, mem_limit=514221656, cur_mem_hold=130023424, attr_.tenant_id_=1) [2024-04-24 10:00:48.895411] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [861710][T1_ReplaySrv2][T1][Y0-0000000000000000-0-0] [lt=82] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924034409607, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924048894751, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:48.895907] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [874706][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=185] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:769, free_obj_hold_cnt:74, allocator used:3351768, allocator total:3466624}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:72, free_obj_hold_cnt:32, allocator used:6821464, allocator total:6821464}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():154752, full allocator used:1600, full allocator total:220288}, tablets_mem=10508376, tablets_mem_limit=257698020, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:23, free_obj_hold_cnt:163, allocator used:369024, allocator total:392448}, pending_cnt=0, wait_gc_count=0, tablet count=841) [2024-04-24 10:00:48.896209] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [861461][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=134] skip inspect bad block(last_check_time=1713875167993818, last_macro_idx=-1) [2024-04-24 10:00:48.902336] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59CA-0-0] [lt=64] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=11, this=0x14b0c0ef7470, extra_info="") [2024-04-24 10:00:48.902470] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59CA-0-0] [lt=98] [PALF STAT GROUP LOG INFO](palf_id=1002, self="10.0.0.3:2882", role="FOLLOWER", total_group_log_cnt=5, avg_log_batch_cnt=0, total_group_log_size=1521, avg_group_log_size=304) [2024-04-24 10:00:48.902568] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59CA-0-0] [lt=92] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="10.0.0.3:2882", committed size=1848) [2024-04-24 10:00:48.902929] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59CA-0-0] [lt=183] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=82, this=0x14b0c0ef76a0, extra_info="palf_id:1002") [2024-04-24 10:00:48.903013] INFO [PALF] read (log_cache.cpp:100) [874558][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=183] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1848, hit_cnt=5, read_cnt=5, hit rate=1.000000000000000000e+00) [2024-04-24 10:00:48.903080] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59CA-0-0] [lt=130] [PALF STAT LOG LIFE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=223976, this=0x14b0c0ef78d0, extra_info="palf_id:1002") [2024-04-24 10:00:48.903173] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59CA-0-0] [lt=82] [PALF STAT LOG TASK TIME](palf_id=1002, self="10.0.0.3:2882", total_slide_log_cnt=5, avg_log_gen_to_freeze_time=18, avg_log_gen_to_submit_time=122, avg_log_submit_to_flush_time=13128, avg_log_submit_to_slide_time=223853) [2024-04-24 10:00:48.903596] INFO [LIB] stat (utility.h:1197) [874557][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=78] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=1674, this=0x14b1312048f8, extra_info="") [2024-04-24 10:00:48.903774] INFO [LIB] stat (utility.h:1197) [874557][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=165] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=222, this=0x14b131204b28, extra_info="") [2024-04-24 10:00:48.903711] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59CA-0-0] [lt=529][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=299916, log_task={header:{begin_lsn:{lsn:40358489}, end_lsn:{lsn:40358611}, log_id:299916, min_scn:{val:1713924048598120000, v:0}, max_scn:{val:1713924048598120000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:40358367}, prev_proposal_id:5, committed_end_lsn:{lsn:40358489}, data_checksum:1715074492, accum_checksum:2542134586, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924048601859, freeze_ts:1713924048601920, submit_ts:1713924048601975, flushed_ts:1713924048618546, gen_to_freeze cost time:61, gen_to_submit cost time:116, submit_to_flush cost time:16571}, fs_cb_begin_ts=1713924048902796, log_life_time=300937) [2024-04-24 10:00:48.903911] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=95] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.904002] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=66] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.904092] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=80] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.910473] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:788) [2316862][T1004_PsCacheEv][T1004][Y0-0000000000000000-0-0] [lt=114] ps cache evict(tenant_id=1004, stmt_id_map_.size()=81, stmt_info_map_.size()=81, op.get_used_size()=251548, get_mem_high()=260919180, expired_stmt_ids.count()=0, closed_stmt_ids.count()=81) [2024-04-24 10:00:48.910790] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=136][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048725291000, v:0}, log_ts:{val:1713924048725291000, v:0}, lsn:{lsn:298521009}, submit_ts:1713924048725291}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:548731}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=185486, ctx_lock_wait_time=7) [2024-04-24 10:00:48.911667] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=117] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8896978944, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:48.912655] INFO [SERVER] do_batch_update_ (ob_tablet_table_updater.cpp:645) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F4-0-0] [lt=92] REPORT: batch update tablets finished(ret=0, ret="OB_SUCCESS", replicas.count()=1, cost_time=92008) [2024-04-24 10:00:48.915224] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=95] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-24 10:00:48.928036] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=111][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.934668] INFO eloop_run (eloop.c:143) [861404][pnio2][T0][Y0-0000000000000000-0-0] [lt=104] PNIO [ratelimit] time: 1713924048934626, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007665, add_bytes: 0 [2024-04-24 10:00:48.942835] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [862266][T1002_LSArchive][T1002][YB420A000003-000616C2AE58F0E7-0-0] [lt=97] gc stale ls task succ [2024-04-24 10:00:48.945741] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=130] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.951676] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:172) [862239][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=118] query dest_info_ succ(dest_info=[], archive_dest_changed=false) [2024-04-24 10:00:48.951810] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:632) [862239][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=109] finish to resize log external storage handler(current_ts=1713924048951791, tenant_max_cpu=1, valid_ls_count=0, other_ls_count=0, new_concurrency=0) [2024-04-24 10:00:48.953705] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=107] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:48.957223] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B0-0-0] [lt=94][errcode=-4023] tx is 2pc logging(this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:48.957476] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B0-0-0] [lt=245][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:48.957652] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B0-0-0] [lt=168][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176913}) [2024-04-24 10:00:48.957744] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B0-0-0] [lt=85][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924048952662, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:48.957821] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B0-0-0] [lt=62][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924048952662, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:48.962849] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=99][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:48.976088] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=76] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:48.976183] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=86] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:48.978139] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=88] no need to update min/max version(new_version={val:1713924048531025000, v:0}, min_version={val:1713924048531025000, v:0}, max_version={val:1713924048531025000, v:0}) [2024-04-24 10:00:48.995096] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2720) [861318][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4002] get invalid Ethernet speed, use default(devname="eth0") [2024-04-24 10:00:48.995211] WDIAG [SERVER] runTimerTask (ob_server.cpp:3239) [861318][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4002] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4002, ret="OB_INVALID_ARGUMENT") [2024-04-24 10:00:48.999856] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [862447][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=101] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:49.003427] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=1592] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.003540] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=97] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.003701] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=146] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.017720] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [874526][T1004_TenantWea][T1003][YB420A000003-000616C2B8CCACF0-0-0] [lt=81] current clock is (clock_=0, max_seq_=0, mem_limit=264912214, cur_mem_hold=121634816, attr_.tenant_id_=1003) [2024-04-24 10:00:49.028263] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=95][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.029619] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=358] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.037221] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=212] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-04-24 10:00:49.046009] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=82] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.048558] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [861771][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=81] current monitor number(seq_=-1) [2024-04-24 10:00:49.053928] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=107] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.054626] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [1559406][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262D-0-0] [lt=140][errcode=-4023] tx is 2pc logging(this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:49.054961] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [1559406][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262D-0-0] [lt=325][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:49.055169] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [1559406][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262D-0-0] [lt=201][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338713}) [2024-04-24 10:00:49.055261] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [1559406][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262D-0-0] [lt=84][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924049050096, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.055342] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [1559406][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262D-0-0] [lt=69][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924049050096, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.059035] INFO [STORAGE.TRANS] retransmit_downstream_msg_ (ob_two_phase_upstream_committer.cpp:311) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=86] unresponded participant(i=1, *this={this:0x14b1481c0010, ref:6, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048664959], ctx_create_time:1713924045155303}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79589464}, {lsn:79618713}], redo_log_no:2, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:6, data_.length():1313}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048664122000, v:0}, trans_type:2, next_log_entry_no:4, max_applied_log_ts:{val:1713924048664122000, v:0}, max_applying_log_ts:{val:1713924048664122000, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:1713924048659397, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:79620195}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048272353001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877443458344 max_table_version=1713877467338312 trans_version={val:1713924048664122000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1 callback_alloc_count=134 callback_free_count=0 checksum=141527899 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=158 redo_sync_succ_count=158 redo_sync_fail_count=0 main_list_length=158 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=158, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924048272353001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048665345, block_frozen_memtable:null}) [2024-04-24 10:00:49.059323] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=192] handle timeout(ret=0, *this={this:0x14b1481c0010, ref:7, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048664959], ctx_create_time:1713924045155303}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79589464}, {lsn:79618713}], redo_log_no:2, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:6, data_.length():1313}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048664122000, v:0}, trans_type:2, next_log_entry_no:4, max_applied_log_ts:{val:1713924048664122000, v:0}, max_applying_log_ts:{val:1713924048664122000, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:1713924048659397, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:79620195}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048272353001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877443458344 max_table_version=1713877467338312 trans_version={val:1713924048664122000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1 callback_alloc_count=134 callback_free_count=0 checksum=141527899 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=158 redo_sync_succ_count=158 redo_sync_fail_count=0 main_list_length=158 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=158, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924048272353001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048665345, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=100000) [2024-04-24 10:00:49.059470] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=136] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:49.059796] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=121] handle timeout(ret=0, *this={this:0x14b055e1e050, ref:4, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924048589222}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88120890}], redo_log_no:1, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048658150001, v:0}, trans_type:2, next_log_entry_no:3, max_applied_log_ts:{val:1713924048658150001, v:0}, max_applying_log_ts:{val:1713924048658150001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:88121631}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048588043001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=0 trans_version={val:1713924048658150001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=1 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924048588043001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048589222, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=100000) [2024-04-24 10:00:49.060059] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=256] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:49.060686] INFO [STORAGE.TRANS] handle_2pc_commit_request_impl_ (ob_two_phase_downstream_committer.cpp:420) [2320947][T1004_L0_G0][T1004][Y0-0000000000000000-0-0] [lt=110] committer is under logging(ret=0, *this={this:0x14b055e1e050, ref:4, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924048589222}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88120890}], redo_log_no:1, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048658150001, v:0}, trans_type:2, next_log_entry_no:3, max_applied_log_ts:{val:1713924048658150001, v:0}, max_applying_log_ts:{val:1713924048658150001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:88121631}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048588043001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=0 trans_version={val:1713924048658150001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=1 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924048588043001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048589222, block_frozen_memtable:null}) [2024-04-24 10:00:49.061101] INFO [STORAGE] exec_wait (ob_tmp_file_cache.cpp:1531) [1556243][T1004_ComTimerT][T1004][Y0-0000000000000000-0-0] [lt=115] succeed to do one round of tmp block io(ret=-4023, loop_nums=0, wait_io_cnt=0, washing_count=0, block_cache_num=0, page_cache_num=0) [2024-04-24 10:00:49.061362] INFO [STORAGE.TRANS] handle_2pc_commit_request_impl_ (ob_two_phase_downstream_committer.cpp:420) [2319281][T1004_L0_G0][T1004][Y0-0000000000000000-0-0] [lt=78] committer is under logging(ret=0, *this={this:0x14b055e1e050, ref:4, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924048589222}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88120890}], redo_log_no:1, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048658150001, v:0}, trans_type:2, next_log_entry_no:3, max_applied_log_ts:{val:1713924048658150001, v:0}, max_applying_log_ts:{val:1713924048658150001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:88121631}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048588043001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=0 trans_version={val:1713924048658150001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=1 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924048588043001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048589222, block_frozen_memtable:null}) [2024-04-24 10:00:49.061758] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [861690][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=96] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-04-24 10:00:49.061833] INFO [STORAGE.TRANS] print_statistics (ob_xa_define.cpp:467) [874732][T1003_ObXAHbWor][T1003][Y0-0000000000000000-0-0] [lt=128] xa statistics(total_active_xa_ctx_count=0, total_standby_clearup_count=0, total_success_xa_start=0, total_failure_xa_start=0, total_success_xa_prepare=0, total_failure_xa_prepare=0, total_success_xa_1pc_commit=0, total_failure_xa_1pc_commit=0, total_success_xa_2pc_commit=0, total_failure_xa_2pc_commit=0, total_failure_xa_rollback=0, total_success_dblink_promotion=0, total_failure_dblink_promotion=0, total_success_dblink=0, total_failure_dblink=0) [2024-04-24 10:00:49.061934] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [861690][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=91] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-04-24 10:00:49.062017] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [861690][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=76] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=9766) [2024-04-24 10:00:49.062994] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=176][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.074037] INFO eloop_run (eloop.c:143) [861399][pnio1][T0][Y0-0000000000000000-0-0] [lt=140] PNIO [ratelimit] time: 1713924049074028, bytes: 14676288946, bw: 0.493096 MB/s, add_ts: 1007545, add_bytes: 520950 [2024-04-24 10:00:49.076917] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=83] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.077038] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=81] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.089237] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [862344][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=88] ====== tenant freeze timer task ====== [2024-04-24 10:00:49.089407] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [862344][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=120] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=96468992, ctx.total_memstore_used_=130023424, ctx.total_memstore_hold_=130023424, memstore_freeze_trigger_limit_=107374180, tenant_id=1001) [2024-04-24 10:00:49.089559] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [862344][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=140] [TenantFreezer] oldest frozen memtable(list="") [2024-04-24 10:00:49.103708] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=121] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.104118] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=126] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.104200] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=71] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.112587] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=108] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8896978944, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:49.128526] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=97][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.129937] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=293] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.141743] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [874742][T1003_LSArchive][T1003][YB420A000003-000616C2B608F0E7-0-0] [lt=86] gc stale ls task succ [2024-04-24 10:00:49.146184] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=90] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.154052] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=106] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.157679] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B1-0-0] [lt=156][errcode=-4023] tx is 2pc logging(this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:49.157857] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B1-0-0] [lt=165][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:49.158133] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B1-0-0] [lt=265][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176913}) [2024-04-24 10:00:49.158215] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B1-0-0] [lt=64][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924049153103, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.158317] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B1-0-0] [lt=81][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924049153103, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.163699] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=120][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.176556] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:259) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=72] readonly requests are active(active_readonly_request_count=3) [2024-04-24 10:00:49.176688] INFO [STORAGE.TRANS] dump_readonly_request (ob_trans_ctx_mgr_v4.cpp:1622) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=122] readonly request is running(trace_id=YB420A000003-000616C2B72929FF-0-0) [2024-04-24 10:00:49.176806] INFO [STORAGE.TRANS] dump_readonly_request (ob_trans_ctx_mgr_v4.cpp:1622) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=89] readonly request is running(trace_id=YB420A000003-000616C2BA89B2C6-0-0) [2024-04-24 10:00:49.177014] INFO [STORAGE.TRANS] dump_readonly_request (ob_trans_ctx_mgr_v4.cpp:1622) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=186] readonly request is running(trace_id=YB420A000003-000616C2ABC99E0B-0-0) [2024-04-24 10:00:49.177105] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=69] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.177201] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=88] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.180776] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [874736][T1003_TenantWea][T1003][YB420A000003-000616C2B5FCA114-0-0] [lt=102] current clock is (clock_=0, max_seq_=0, mem_limit=264912214, cur_mem_hold=121634816, attr_.tenant_id_=1003) [2024-04-24 10:00:49.204171] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=86] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.204327] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=134] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.204415] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=77] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.220596] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861496][RpcIO][T0][Y0-0000000000000000-0-0] [lt=119] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:49.220966] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [874754][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=105] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:49.221292] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861497][RpcIO][T0][Y0-0000000000000000-0-0] [lt=73] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:49.221390] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861495][RpcIO][T0][Y0-0000000000000000-0-0] [lt=61] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:49.222250] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861498][RpcIO][T0][Y0-0000000000000000-0-0] [lt=85] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:49.228673] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=90][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.229380] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=57] ====== checkpoint timer task ====== [2024-04-24 10:00:49.229558] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=109] get_max_applied_scn(scn={val:1713924048573312000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:5, palf_committed_end_lsn_:{lsn:67159339}, last_check_scn_:{val:1713924048836268000, v:0}, max_applied_cb_scn_:{val:1713924048573312000, v:0}}) [2024-04-24 10:00:49.229734] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1018) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=164] get_min_unreplayed_log_info in skip state(lsn={lsn:64863204}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713921656489962, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:64863204}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:64863204}, end_lsn:{lsn:64863204}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x14b1470b1470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-04-24 10:00:49.230001] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=75] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.230909] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=207] succ to get rec scn(*this={this:0x14b13a604030, ls_id:{id:1}, tenant_id:1002, state:"L_WORKING", total_tx_ctx_count:0, active_tx_count:0, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1713895203755260001, v:0}, last_push_gc_task_ts_:1713895209136039, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:602}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-24 10:00:49.231042] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=104] tx ctx memtable get rec scn(this={ObITable:{this:0x14b14714c080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1713895204842104, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x14b14714c080, snapshot_version:{val:1713895204842104, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-04-24 10:00:49.231156] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=90] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-04-24 10:00:49.231421] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=69] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1713895310268028001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60379}, rec_scn:{val:1713895310268028001, v:0}}, }, *this={this:0x14b147070510, is_inited:true, freezing_scn:{val:1713877162774754000, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895310268028001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924044057788000, v:0}, tablet_change_checkpoint_scn:{val:1713875507773116000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924056491232000, latest_log_ts:{val:1713924039886464001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713890395635598001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:5, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:49.231556] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=106] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x14b147070510, is_inited:true, freezing_scn:{val:1713877162774754000, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895310268028001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924044057788000, v:0}, tablet_change_checkpoint_scn:{val:1713875507773116000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924056491232000, latest_log_ts:{val:1713924039886464001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713890395635598001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:5, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}, min_rec_scn={val:1713895310268028001, v:0}, ls_id_={id:1}) [2024-04-24 10:00:49.231778] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=206] get rec log scn(service_type_=0, rec_log_ts={val:1713924039886464001, v:0}) [2024-04-24 10:00:49.231978] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=192] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-04-24 10:00:49.232089] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=75] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713895310268028001, v:0}, checkpoint_scn_in_ls_meta={val:1713895310268028001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-04-24 10:00:49.232191] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=79][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, curr_base_lsn={lsn:0}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2024-04-24 10:00:49.232318] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=118] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2024-04-24 10:00:49.232441] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=74] get_max_applied_scn(scn={val:1713924048573312001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:7, palf_committed_end_lsn_:{lsn:1002196895}, last_check_scn_:{val:1713924048914830000, v:0}, max_applied_cb_scn_:{val:1713924048573312001, v:0}}) [2024-04-24 10:00:49.232546] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=86] get_min_unreplayed_log_info(lsn={lsn:951799104}, scn={val:1713900501780579001, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713921656899305, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:4063, ret_code_:0}}, next_to_submit_lsn_:{lsn:951799104}, next_to_submit_scn_:{val:1713900501780579001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1713875572267468000, v:0}, iterator_:{iterator_impl:{buf_:0x14b0ec177000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:951799104}, end_lsn:{lsn:951799104}, read_buf:{buf_len_:8192, buf_:0x14b0ec177000}, block_size:67104768, log_storage_:0x14b1336f1470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-04-24 10:00:49.234128] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=162] succ to get rec scn(*this={this:0x14b13a6b6030, ls_id:{id:1001}, tenant_id:1002, state:"L_WORKING", total_tx_ctx_count:0, active_tx_count:0, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1713890378240272001, v:0}, last_push_gc_task_ts_:1713892170603366, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:732}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-24 10:00:49.234253] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=118] tx ctx memtable get rec scn(this={ObITable:{this:0x14b14714ce90, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1713892170607112, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x14b14714ce90, snapshot_version:{val:1713892170607112, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-04-24 10:00:49.234371] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=101] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2024-04-24 10:00:49.234779] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=110] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1713895385940970001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:200589}, rec_scn:{val:1713895385940970001, v:0}}, }, *this={this:0x14b13438c510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1713895385940970001, v:0}, clog_base_lsn:{lsn:335523840}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924044057788000, v:0}, tablet_change_checkpoint_scn:{val:1713890381649962000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:7, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:49.235002] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=204] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x14b13438c510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1713895385940970001, v:0}, clog_base_lsn:{lsn:335523840}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924044057788000, v:0}, tablet_change_checkpoint_scn:{val:1713890381649962000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:7, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}, min_rec_scn={val:1713895385940970001, v:0}, ls_id_={id:1001}) [2024-04-24 10:00:49.235192] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=175] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713895385940970001, v:0}, checkpoint_scn_in_ls_meta={val:1713895385940970001, v:0}, ls_id={id:1001}, service_type="TRANS_SERVICE") [2024-04-24 10:00:49.235299] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=86][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, curr_base_lsn={lsn:335523840}, new_base_lsn={lsn:335523840}, lsn={lsn:335523840}) [2024-04-24 10:00:49.235385] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=76] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:335523840}, ls->get_ls_id()={id:1001}) [2024-04-24 10:00:49.235478] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=69] get_max_applied_scn(scn={val:1713875572331196000, v:0}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1713875572331196000, v:0}}) [2024-04-24 10:00:49.235563] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=73] get_min_unreplayed_log_info(lsn={lsn:165102580}, scn={val:1713924048554997001, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713924048885617, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:165102580}, next_to_submit_scn_:{val:1713924048554997001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1713875572331196000, v:0}, iterator_:{iterator_impl:{buf_:0x14b1144cd000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:165102458}, end_lsn:{lsn:165102580}, read_buf:{buf_len_:24576, buf_:0x14b1144cd000}, block_size:67104768, log_storage_:0x14b13adc9470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713924048554997000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713924048554997000, v:0}, data_checksum:3348837161, flag:1}}, init_mode_version:0, accumulate_checksum:144396744, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713924048554997000, v:0}}}}}) [2024-04-24 10:00:49.236667] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=113] succ to get rec scn(*this={this:0x14b129004030, ls_id:{id:1002}, tenant_id:1002, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1713890378211605001, v:0}, last_push_gc_task_ts_:1713892170605151, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:734}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-24 10:00:49.236761] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=86] tx ctx memtable get rec scn(this={ObITable:{this:0x14b14714dca0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1713892170644650, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x14b14714dca0, snapshot_version:{val:1713892170644650, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-04-24 10:00:49.236906] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=89] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2024-04-24 10:00:49.237216] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=60] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1713895298278308001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:200876}, rec_scn:{val:1713895298278308001, v:0}}, }, *this={this:0x14b133736510, is_inited:true, freezing_scn:{val:1713890363233517001, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_create_status:1, clog_checkpoint_scn:{val:1713895298278308001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924044057788000, v:0}, tablet_change_checkpoint_scn:{val:1713890384892147000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:2, proposal_id:0, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:1942234752, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:49.237382] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=145] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x14b133736510, is_inited:true, freezing_scn:{val:1713890363233517001, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_create_status:1, clog_checkpoint_scn:{val:1713895298278308001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924044057788000, v:0}, tablet_change_checkpoint_scn:{val:1713890384892147000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:2, proposal_id:0, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:1942234752, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}, min_rec_scn={val:1713895298278308001, v:0}, ls_id_={id:1002}) [2024-04-24 10:00:49.237510] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=112] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713895298278308001, v:0}, checkpoint_scn_in_ls_meta={val:1713895298278308001, v:0}, ls_id={id:1002}, service_type="TRANS_SERVICE") [2024-04-24 10:00:49.237561] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=86] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-04-24 10:00:49.237604] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=82][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, curr_base_lsn={lsn:0}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2024-04-24 10:00:49.237710] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=96] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2024-04-24 10:00:49.237786] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [862261][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=66] succeed to update_clog_checkpoint(ret=0, ls_cnt=3) [2024-04-24 10:00:49.240969] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=110] ====== checkpoint timer task ====== [2024-04-24 10:00:49.241128] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=83] get_max_applied_scn(scn={val:1713924048510522000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:5, palf_committed_end_lsn_:{lsn:79624410}, last_check_scn_:{val:1713924048839739001, v:0}, max_applied_cb_scn_:{val:1713924048510522000, v:0}}) [2024-04-24 10:00:49.241223] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1018) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=89] get_min_unreplayed_log_info in skip state(lsn={lsn:69192977}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713921656897811, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:69192977}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:69192977}, end_lsn:{lsn:69192977}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x14b0efb2f470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-04-24 10:00:49.242741] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=103] succ to get rec scn(*this={this:0x14b0db804030, ls_id:{id:1}, tenant_id:1004, state:"L_WORKING", total_tx_ctx_count:298, active_tx_count:3, total_active_readonly_request_count:3, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():295, max_wait_ckpt_ts_:{val:1713924045912367001, v:0}, last_push_gc_task_ts_:1713923772384693, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:602}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-24 10:00:49.242833] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=86] tx ctx memtable get rec scn(this={ObITable:{this:0x14b0efbc0080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1713895201195017, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x14b0efbc0080, snapshot_version:{val:1713895201195017, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:1713923771476976001, v:0}) [2024-04-24 10:00:49.242952] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=111] rec_scn of ObLockMemtable is (rec_scn_={val:1713923803347774003, v:0}, flushed_scn_={val:1713880213703915001, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:1713880213703915001, v:0}, max_committed_scn_={val:1713924045912367001, v:0}, is_frozen_=false, ls_id_={id:1}) [2024-04-24 10:00:49.243197] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=95] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1713895222371319002, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60379}, rec_scn:{val:1713895222371319002, v:0}}, }, *this={this:0x14b0efaf8510, is_inited:true, freezing_scn:{val:1713879137201785000, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895205050766000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924043670180000, v:0}, tablet_change_checkpoint_scn:{val:1713877445609286000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924058572929000, latest_log_ts:{val:1713924041908950001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713919200442801001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:5, palf_env_:0x14b176ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:49.243321] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=109] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x14b0efaf81b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x14b0efafdcd0, table_type:1, memtable_head:3, memtable_tail:4, t3m:0x14b13ecb2030, tables:[null, null, null, 0x14b0efba16a0, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:1713895203828281, tx_data_table:0x14b0efaff1d0, ls_tablet_svr:0x14b0efaf8190, slice_allocator:0x14b0efaff210}, min_rec_scn={val:1713895205050766000, v:0}, ls_id_={id:1}) [2024-04-24 10:00:49.243417] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=77] get rec log scn(service_type_=0, rec_log_ts={val:1713924041908950001, v:0}) [2024-04-24 10:00:49.243525] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=93] get rec log scn(service_type_=1, rec_log_ts={val:1713919200442801001, v:0}) [2024-04-24 10:00:49.243616] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=73] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713895205050766000, v:0}, checkpoint_scn_in_ls_meta={val:1713895205050766000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-04-24 10:00:49.243745] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=106][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, curr_base_lsn={lsn:0}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2024-04-24 10:00:49.243818] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=65] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2024-04-24 10:00:49.243964] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=131] get_max_applied_scn(scn={val:1713924048588043000, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:7, palf_committed_end_lsn_:{lsn:88121908}, last_check_scn_:{val:1713924048823139001, v:0}, max_applied_cb_scn_:{val:1713924048588043000, v:0}}) [2024-04-24 10:00:49.244055] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=80] get_min_unreplayed_log_info(lsn={lsn:82126713}, scn={val:1713900501167593003, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713921656936517, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:82126713}, next_to_submit_scn_:{val:1713900501167593003, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1713877500257917000, v:0}, iterator_:{iterator_impl:{buf_:0x14b12fd69000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:82126713}, end_lsn:{lsn:82126713}, read_buf:{buf_len_:8192, buf_:0x14b12fd69000}, block_size:67104768, log_storage_:0x14b0c0e31470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-04-24 10:00:49.245606] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=125] succ to get rec scn(*this={this:0x14b0db8b6030, ls_id:{id:1001}, tenant_id:1004, state:"L_WORKING", total_tx_ctx_count:151, active_tx_count:2, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():149, max_wait_ckpt_ts_:{val:1713924045471130001, v:0}, last_push_gc_task_ts_:1713923772386258, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:474}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-24 10:00:49.245720] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=101] tx ctx memtable get rec scn(this={ObITable:{this:0x14b0efbc0e90, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1713881446631656, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x14b0efbc0e90, snapshot_version:{val:1713881446631656, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:1713923772268263001, v:0}) [2024-04-24 10:00:49.245851] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=88] rec_scn of ObLockMemtable is (rec_scn_={val:1713923808203128000, v:0}, flushed_scn_={val:1713880213703915002, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:1713880213703915002, v:0}, max_committed_scn_={val:1713924045471130001, v:0}, is_frozen_=false, ls_id_={id:1001}) [2024-04-24 10:00:49.246033] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=133] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1713924005270741002, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606847570}, rec_scn:{val:1713924005270741002, v:0}}, }, *this={this:0x14b0c0f06510, is_inited:true, freezing_scn:{val:1713879643147421001, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1713895202258988000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924043670180000, v:0}, tablet_change_checkpoint_scn:{val:1713923988969639000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:7, palf_env_:0x14b176ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:2, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:49.246168] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=111] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x14b0c0f061b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x14b0c0f0bcd0, table_type:1, memtable_head:2, memtable_tail:3, t3m:0x14b13ecb2030, tables:[null, null, 0x14b0efba2cc0, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1001}, mini_merge_recycle_commit_versions_ts:1713895204982541, tx_data_table:0x14b0c0f0d1d0, ls_tablet_svr:0x14b0c0f06190, slice_allocator:0x14b0c0f0d210}, min_rec_scn={val:1713895202258988000, v:0}, ls_id_={id:1001}) [2024-04-24 10:00:49.246306] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=121] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713895202258988000, v:0}, checkpoint_scn_in_ls_meta={val:1713895202258988000, v:0}, ls_id={id:1001}, service_type="TRANS_SERVICE") [2024-04-24 10:00:49.246461] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=145][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, curr_base_lsn={lsn:0}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2024-04-24 10:00:49.246388] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=102] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.246557] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=88] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2024-04-24 10:00:49.246670] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=96] get_max_applied_scn(scn={val:1713877500304026000, v:0}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1713877500304026000, v:0}}) [2024-04-24 10:00:49.246755] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=76] get_min_unreplayed_log_info(lsn={lsn:40358611}, scn={val:1713924048598120001, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713924048902805, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:2137654, ret_code_:0}}, next_to_submit_lsn_:{lsn:40358611}, next_to_submit_scn_:{val:1713924048598120001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1713877500304026000, v:0}, iterator_:{iterator_impl:{buf_:0x14b07d217000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:40358489}, end_lsn:{lsn:40358611}, read_buf:{buf_len_:53248, buf_:0x14b07d217000}, block_size:67104768, log_storage_:0x14b0c0efd470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713924048598120000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713924048598120000, v:0}, data_checksum:3348837161, flag:1}}, init_mode_version:0, accumulate_checksum:2542134586, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713924048598120000, v:0}}}}}) [2024-04-24 10:00:49.248104] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=83] succ to get rec scn(*this={this:0x14b0bfc04030, ls_id:{id:1002}, tenant_id:1004, state:"F_WORKING", total_tx_ctx_count:145, active_tx_count:0, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():145, max_wait_ckpt_ts_:{val:1713924044495590001, v:0}, last_push_gc_task_ts_:1713923772387529, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:457}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-24 10:00:49.248200] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=91] tx ctx memtable get rec scn(this={ObITable:{this:0x14b0efbc1ca0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1713923772391520, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x14b0efbc1ca0, snapshot_version:{val:1713923772391520, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:1713923773982428003, v:0}) [2024-04-24 10:00:49.248300] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=92] rec_scn of ObLockMemtable is (rec_scn_={val:1713923803363759001, v:0}, flushed_scn_={val:1713880196410549002, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:1713880196410549002, v:0}, max_committed_scn_={val:1713924044495590001, v:0}, is_frozen_=false, ls_id_={id:1002}) [2024-04-24 10:00:49.248475] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=109] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1713923993909177001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606847566}, rec_scn:{val:1713923993909177001, v:0}}, }, *this={this:0x14b0c0eb6510, is_inited:true, freezing_scn:{val:1713923772239168001, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1002}, ls_create_status:1, clog_checkpoint_scn:{val:1713923772377160001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924043670180000, v:0}, tablet_change_checkpoint_scn:{val:1713923978662610000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:2, proposal_id:0, palf_env_:0x14b176ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:2, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:49.248613] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=115] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x14b0c0eb61b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x14b0c0ebbcd0, table_type:1, memtable_head:3, memtable_tail:4, t3m:0x14b13ecb2030, tables:[null, null, null, 0x14b0efba0080, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1002}, mini_merge_recycle_commit_versions_ts:1713923773084848, tx_data_table:0x14b0c0ebd1d0, ls_tablet_svr:0x14b0c0eb6190, slice_allocator:0x14b0c0ebd210}, min_rec_scn={val:1713923772377160001, v:0}, ls_id_={id:1002}) [2024-04-24 10:00:49.248753] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=127] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713923772377160001, v:0}, checkpoint_scn_in_ls_meta={val:1713923772377160001, v:0}, ls_id={id:1002}, service_type="TRANS_SERVICE") [2024-04-24 10:00:49.248836] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=77][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, curr_base_lsn={lsn:0}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2024-04-24 10:00:49.248957] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=111] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2024-04-24 10:00:49.249047] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [874527][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=84] succeed to update_clog_checkpoint(ret=0, ls_cnt=3) [2024-04-24 10:00:49.254238] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=105] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.255020] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [879578][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262E-0-0] [lt=82][errcode=-4023] tx is 2pc logging(this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:49.255279] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [879578][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262E-0-0] [lt=244][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:49.255455] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [879578][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262E-0-0] [lt=161][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338713}) [2024-04-24 10:00:49.255556] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [879578][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262E-0-0] [lt=89][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924049250430, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.255747] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [879578][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262E-0-0] [lt=171][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924049250430, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.258044] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1457) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B39C6513-0-0] [lt=117] receive register_req from acceptor, ignore(self="10.0.0.3:2882", palf_id=1001, child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924049252976, keepalive_ts:-1}) [2024-04-24 10:00:49.258145] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1476) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B39C6513-0-0] [lt=93] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="10.0.0.3:2882", child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924049252976, keepalive_ts:-1}, is_to_leader=true) [2024-04-24 10:00:49.260834] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [874525][T1004_ReqMemEvi][T1004][Y0-0000000000000000-0-0] [lt=90] sql audit evict task end(request_manager_->get_tenant_id()=1004, evict_high_mem_level=278313880, evict_high_size_level=90000, evict_batch_count=0, elapse_time=8, size_used=80563, mem_used=131023872) [2024-04-24 10:00:49.263968] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=79][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.269165] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [862437][T1001_LSArchive][T1001][YB420A000003-000616C2AEF8F0E7-0-0] [lt=87] gc stale ls task succ [2024-04-24 10:00:49.276856] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=81] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.277018] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=154] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.301781] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [874706][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=135] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:769, free_obj_hold_cnt:74, allocator used:3351768, allocator total:3466624}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:72, free_obj_hold_cnt:32, allocator used:6821464, allocator total:6821464}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():154752, full allocator used:1600, full allocator total:220288}, tablets_mem=10508376, tablets_mem_limit=257698020, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:23, free_obj_hold_cnt:163, allocator used:369024, allocator total:392448}, pending_cnt=0, wait_gc_count=0, tablet count=841) [2024-04-24 10:00:49.302218] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [862436][T1001_TabletShe][T1001][Y0-0000000000000000-0-0] [lt=116] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-04-24 10:00:49.302392] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [862436][T1001_TabletShe][T1001][Y0-0000000000000000-0-0] [lt=95] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=9730) [2024-04-24 10:00:49.304366] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=101] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.304481] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=83] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.304554] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=61] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.313430] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=115] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8894881792, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:49.325338] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=94] LogLoopThread round_cost_time(us)(round_cost_time=101) [2024-04-24 10:00:49.328932] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=122][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.330221] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=81] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.344104] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [862424][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=159] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=1, total_destructed_detector=1, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x14b15a1e42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-04-24 10:00:49.346611] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=161] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.349141] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [874735][T1003_ReqMemEvi][T1003][Y0-0000000000000000-0-0] [lt=109] sql audit evict task end(request_manager_->get_tenant_id()=1003, evict_high_mem_level=19327352, evict_high_size_level=90000, evict_batch_count=0, elapse_time=7, size_used=8111, mem_used=14558208) [2024-04-24 10:00:49.354452] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=97] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.358220] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2325187][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B2-0-0] [lt=94][errcode=-4023] tx is 2pc logging(this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:49.358451] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2325187][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B2-0-0] [lt=203][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:49.358575] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2325187][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B2-0-0] [lt=114][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176913}) [2024-04-24 10:00:49.358645] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2325187][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B2-0-0] [lt=56][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924049353735, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.358727] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2325187][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B2-0-0] [lt=63][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924049353735, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.360480] INFO [STORAGE.TRANS] retransmit_downstream_msg_ (ob_two_phase_upstream_committer.cpp:311) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=115] unresponded participant(i=1, *this={this:0x14b1481c0010, ref:6, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048664959], ctx_create_time:1713924045155303}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79589464}, {lsn:79618713}], redo_log_no:2, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:6, data_.length():1313}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048664122000, v:0}, trans_type:2, next_log_entry_no:4, max_applied_log_ts:{val:1713924048664122000, v:0}, max_applying_log_ts:{val:1713924048664122000, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:1713924048659397, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:79620195}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048272353001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877443458344 max_table_version=1713877467338312 trans_version={val:1713924048664122000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1 callback_alloc_count=134 callback_free_count=0 checksum=141527899 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=158 redo_sync_succ_count=158 redo_sync_fail_count=0 main_list_length=158 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=158, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924048272353001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048665345, block_frozen_memtable:null}) [2024-04-24 10:00:49.360949] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=282] handle timeout(ret=0, *this={this:0x14b1481c0010, ref:7, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048664959], ctx_create_time:1713924045155303}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79589464}, {lsn:79618713}], redo_log_no:2, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:6, data_.length():1313}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048664122000, v:0}, trans_type:2, next_log_entry_no:4, max_applied_log_ts:{val:1713924048664122000, v:0}, max_applying_log_ts:{val:1713924048664122000, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:1713924048659397, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:79620195}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048272353001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877443458344 max_table_version=1713877467338312 trans_version={val:1713924048664122000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1 callback_alloc_count=134 callback_free_count=0 checksum=141527899 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=158 redo_sync_succ_count=158 redo_sync_fail_count=0 main_list_length=158 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=158, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924048272353001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048665345, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=200000) [2024-04-24 10:00:49.361136] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=181] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:49.361355] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=132] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: MICRO_BLOCK_CACHE_IO, mode: read, size: 4096.00, iops: 11.91, rt: 768.67) [2024-04-24 10:00:49.361355] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=100] handle timeout(ret=0, *this={this:0x14b055e1e050, ref:4, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924048589222}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88120890}], redo_log_no:1, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048658150001, v:0}, trans_type:2, next_log_entry_no:3, max_applied_log_ts:{val:1713924048658150001, v:0}, max_applying_log_ts:{val:1713924048658150001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:88121631}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048588043001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=0 trans_version={val:1713924048658150001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=1 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924048588043001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048589222, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=200000) [2024-04-24 10:00:49.361464] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=77] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:49.361488] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=126] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:49.361926] INFO [STORAGE.TRANS] handle_2pc_commit_request_impl_ (ob_two_phase_downstream_committer.cpp:420) [2319281][T1004_L0_G0][T1004][Y0-0000000000000000-0-0] [lt=61] committer is under logging(ret=0, *this={this:0x14b055e1e050, ref:4, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924048589222}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88120890}], redo_log_no:1, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048658150001, v:0}, trans_type:2, next_log_entry_no:3, max_applied_log_ts:{val:1713924048658150001, v:0}, max_applying_log_ts:{val:1713924048658150001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:88121631}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048588043001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=0 trans_version={val:1713924048658150001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=1 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924048588043001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048589222, block_frozen_memtable:null}) [2024-04-24 10:00:49.362053] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=83] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 158.00) [2024-04-24 10:00:49.362160] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=81] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:49.362379] INFO [SQL.JO] compute_table_location (ob_join_order.cpp:278) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1E98-0-0] [lt=113] succeed to calculate base table sharding info(table_id=18446744073709551614, ref_table_id=104, is_global_index=false) [2024-04-24 10:00:49.362501] INFO [SQL.JO] init_est_sel_info_for_access_path (ob_join_order.cpp:12189) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1E98-0-0] [lt=101] total rowcount, use statistics(table_meta_info_.table_row_count_=10, table_meta_info_.average_row_size_=2.060000000000000000e+02, table_meta_info_.micro_block_count_=1, table_meta_info_.part_size_=2.060000000000000000e+03) [2024-04-24 10:00:49.362692] INFO [STORAGE.TRANS] handle_2pc_commit_request_impl_ (ob_two_phase_downstream_committer.cpp:420) [2319281][T1004_L0_G0][T1004][Y0-0000000000000000-0-0] [lt=133] committer is under logging(ret=0, *this={this:0x14b055e1e050, ref:4, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924048589222}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88120890}], redo_log_no:1, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048658150001, v:0}, trans_type:2, next_log_entry_no:3, max_applied_log_ts:{val:1713924048658150001, v:0}, max_applying_log_ts:{val:1713924048658150001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:88121631}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048588043001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=0 trans_version={val:1713924048658150001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=1 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924048588043001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048589222, block_frozen_memtable:null}) [2024-04-24 10:00:49.364131] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=77][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.364376] INFO [COMMON] print_io_status (ob_io_manager.cpp:1393) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=93] [IO STATUS SYS](tenant_id=1004, io_status=sys_group_name: SLOG_IO, mode: write, size: 4096.00, iops: 0.99, rt: 3645.00) [2024-04-24 10:00:49.364480] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=79] [IO STATUS CONFIG](tenant_id=1004, ref_cnt=2, io_config={group_num_:0, memory_limit_:11596411700, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:2}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:49.364582] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=84] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-24 10:00:49.364795] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=197] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-24 10:00:49.370327] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:834) [862160][T1002_BalanceEx][T1002][YB420A000003-000616C2B3D9ADA0-0-0] [lt=77] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2024-04-24 10:00:49.370417] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:171) [862160][T1002_BalanceEx][T1002][YB420A000003-000616C2B3D9ADA0-0-0] [lt=81] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2024-04-24 10:00:49.377158] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=82] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.377247] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=79] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.402689] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1E98-0-0] [lt=99][errcode=0] set logical del time(cache_obj->get_logical_del_time()=576361976805, cache_obj->added_lc()=false, cache_obj->get_object_id()=39597, cache_obj->get_tenant_id()=1, lbt()="0x121f36ac 0xc327dd4 0x4e3fdbe 0x4e31e28 0x4e29b67 0x4e21201 0x4e1eb17 0x4e0e175 0xa7ac354 0x12d1fdbc 0x14b1c1d132de 0x14b1c1a44e83") [2024-04-24 10:00:49.404465] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=65] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.404580] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=90] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.404696] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=110] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.405597] INFO [STORAGE.TRANS] get_gts (ob_gts_source.cpp:268) [861910][T1_L0_G0][T1][YB420A000003-000616C2ADA8FD64-0-0] [lt=59] push gts task success(*task={:""}) [2024-04-24 10:00:49.410341] WDIAG get_refactored (ob_hashtable.h:1100) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1E9B-0-0] [lt=80][errcode=-4201] hashtable not init [2024-04-24 10:00:49.410436] WDIAG [SQL.SESSION] get_cursor (ob_sql_session_info.cpp:1439) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1E9B-0-0] [lt=85][errcode=-4016] get cursor info failed(cursor_id=2, get_sessid()=3221540160) [2024-04-24 10:00:49.410961] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1E9C-0-0] [lt=62] rpc statistics(total_trans_msg_count=3, total_batch_msg_count=0) [2024-04-24 10:00:49.413202] INFO [STORAGE.TRANS] amend (ob_tenant_weak_read_server_version_mgr.cpp:192) [861686][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=89] [WRS] tenant has no valid partition on current server, generate weak read snapshot version based on current timestamp(tenant_id=1, epoch_tstamp_=1713924049413024, total_part_count_=1, valid_inner_part_count_=0, valid_user_part_count_=0, version_={val:1713924044902249000, v:0}, base_version_when_no_valid_partition={val:1713924044902249000, v:0}) [2024-04-24 10:00:49.416709] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [861702][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=150] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:49.416935] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=103] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=253, get_gts_with_stc_cnt=17, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=19, try_wait_gts_elapse_cnt=0) [2024-04-24 10:00:49.419783] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [874510][T1004_LockWaitM][T1004][Y0-0000000000000000-0-0] [lt=87] report RowHolderMapper summary info(count=137, bkt_cnt=248) [2024-04-24 10:00:49.429062] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=110][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.430379] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=137] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.439037] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=120] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=990864, replace_num=15728) [2024-04-24 10:00:49.439179] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=134] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-04-24 10:00:49.445041] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861506][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=91] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:49.445829] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861508][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=85] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:49.446180] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861507][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=78] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:49.446504] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861509][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=68] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:49.446770] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=206] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.447714] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861511][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=82] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:49.448270] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861500][BatchIO][T0][Y0-0000000000000000-0-0] [lt=91] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:49.449030] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861502][BatchIO][T0][Y0-0000000000000000-0-0] [lt=75] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:49.449046] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861501][BatchIO][T0][Y0-0000000000000000-0-0] [lt=149] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:49.449178] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861503][BatchIO][T0][Y0-0000000000000000-0-0] [lt=114] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:49.450275] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861504][BatchIO][T0][Y0-0000000000000000-0-0] [lt=114] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:49.450373] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861513][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=93] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:49.454597] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=105] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.464336] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=124][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.471351] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1457) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2B7EC314F-0-0] [lt=124] receive register_req from acceptor, ignore(self="10.0.0.3:2882", palf_id=1, child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924049466708, keepalive_ts:-1}) [2024-04-24 10:00:49.471448] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1476) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2B7EC314F-0-0] [lt=90] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="10.0.0.3:2882", child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924049466708, keepalive_ts:-1}, is_to_leader=true) [2024-04-24 10:00:49.477266] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=91] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.477354] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=75] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.479110] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [861474][Blacklist][T0][Y0-0000000000000000-0-0] [lt=78] blacklist_loop exec finished(cost_time=48, is_enabled=true, send_cnt=0) [2024-04-24 10:00:49.504757] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=77] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.504933] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=151] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.505012] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=71] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.514537] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=153] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8894881792, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:49.520162] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:834) [874425][T1004_BalanceEx][T1004][YB420A000003-000616C2BB69A617-0-0] [lt=97] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2024-04-24 10:00:49.520283] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:171) [874425][T1004_BalanceEx][T1004][YB420A000003-000616C2BB69A617-0-0] [lt=95] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2024-04-24 10:00:49.523437] INFO [CLOG] run1 (ob_garbage_collector.cpp:1338) [862243][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=90] Garbage Collector is running(seq_=4855, gc_interval=10000000) [2024-04-24 10:00:49.523603] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1473) [862243][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=81] self is leader, skip it(ls->get_ls_id()={id:1}) [2024-04-24 10:00:49.523743] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1473) [862243][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=117] self is leader, skip it(ls->get_ls_id()={id:1001}) [2024-04-24 10:00:49.524630] INFO [STORAGE] wait_unpersist_tablet_ids_flushed (ob_tablet_gc_service.cpp:600) [874530][T1004_ObTimer][T1004][Y0-0000000000000000-0-0] [lt=155] [tabletgc] wait tablet flush(rec_scn={val:1713923993909177001, v:0}, decided_scn={val:1713923994199673000, v:0}, retry_times=62, i=4, ls_->get_ls_id()={id:1002}, tablet={this:0x14b078ebbda0, is_inited:true, wash_score:1713924047805202000, hold_ref_cnt:true, ref_cnt:2, version:2, length:0, tablet_addr:[3460-2906-0-2305843609435840512-0](offset=466944,size=559,type=2,seq=0), allocator:null, tablet_meta:{version:1, ls_id:{id:1002}, tablet_id:{id:1152921504606847566}, data_tablet_id:{id:200314}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:1713923993909177001, v:0}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1713924014389092000, v:0}, snapshot_version:1713924006239779000, multi_version_start:1713924006239779000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1713924006239779000, cur_report_version:1713895200302650000, data_checksum:0, row_count:0}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:1713924014025221001, v:0}, ddl_snapshot_version:1713924006239779000, max_sync_storage_schema_version:1713923993578648, max_serialized_medium_scn:0, ddl_execution_id:0, ddl_data_format_version:17180000768, ddl_commit_scn:{val:1713924014389092000, v:0}, mds_checkpoint_scn:{val:1, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1713923993578648}, table_store_addr:{ptr:{this:0x14b078ebc458, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:0}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[3460-2906-0-2305843331336699904-0](offset=458752,size=300,type=2,seq=0), type:"MAJOR", tablet_id:{id:1152921504606847566}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1713924006239779000, v:0}}, snapshot_version:1713924006239779000, max_merge_version:1713924006239779000}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{null}}}, addr:[3460-2906-0-2305843250806067200-0](offset=462848,size=225,type=2,seq=0)}, storage_schema_addr:{ptr:null, addr:[3460-2906-0-2305843170275430625-0](offset=463073,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:4097, compat:1, last_compaction_type:0, wait_check_flag:1, last_medium_scn:1713924006239779000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:7, str:"MAX"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:0, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0}) [2024-04-24 10:00:49.525464] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:272) [862243][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=82] GC check ls in member list, skip it(id={id:1002}, leader="10.0.0.2:2882") [2024-04-24 10:00:49.525589] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1431) [862243][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=106] gc_check_member_list_ cost time(ret=0, time_us=2052) [2024-04-24 10:00:49.525771] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [862243][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=155] execute_gc cost time(ret=0, time_us=7) [2024-04-24 10:00:49.527801] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1626) [862243][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=159] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2024-04-24 10:00:49.529221] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=94][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.529534] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1626) [862243][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=161] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}) [2024-04-24 10:00:49.530537] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=79] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.531454] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1626) [862243][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=74] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}) [2024-04-24 10:00:49.531596] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1648) [862243][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=77] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}]) [2024-04-24 10:00:49.531757] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1648) [862243][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=119] ls status is normal, skip(id={id:1001}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}]) [2024-04-24 10:00:49.531998] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1648) [862243][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=192] ls status is normal, skip(id={id:1002}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}]) [2024-04-24 10:00:49.532169] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [862243][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=107] execute_gc cost time(ret=0, time_us=590) [2024-04-24 10:00:49.532271] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [862243][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=81] ObSafeDestroyHandler start process [2024-04-24 10:00:49.538782] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1044) [861646][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=83] success to update min reserved snapshot(reserved_snapshot=1713922246888571000, duration=1800, snapshot_gc_ts_=1713924046888571000) [2024-04-24 10:00:49.539027] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1071) [861646][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=142] update reserved snapshot finished(cost_ts=251, reserved_snapshot=1713922246888571000) [2024-04-24 10:00:49.546954] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=150] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.554807] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=187] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.556253] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262F-0-0] [lt=85][errcode=-4023] tx is 2pc logging(this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:49.556398] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262F-0-0] [lt=137][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:49.556490] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262F-0-0] [lt=85][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338713}) [2024-04-24 10:00:49.556571] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262F-0-0] [lt=72][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924049551086, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.556710] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC262F-0-0] [lt=120][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924049551086, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.560443] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B3-0-0] [lt=138][errcode=-4023] tx is 2pc logging(this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:49.560698] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B3-0-0] [lt=238][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:49.560922] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B3-0-0] [lt=161][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176913}) [2024-04-24 10:00:49.561064] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B3-0-0] [lt=135][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924049555748, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.561166] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [862313][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=78] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2024-04-24 10:00:49.561228] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B3-0-0] [lt=142][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924049555748, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.563543] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [862259][T1002_ReqMemEvi][T1002][Y0-0000000000000000-0-0] [lt=133] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=16106127, evict_high_size_level=90000, evict_batch_count=0, elapse_time=7, size_used=6645, mem_used=12478464) [2024-04-24 10:00:49.564497] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=56][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.575001] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [862430][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=137] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=90000, evict_batch_count=0, elapse_time=13, size_used=8502, mem_used=14558208) [2024-04-24 10:00:49.577420] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=67] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.577518] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=83] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.578241] INFO [STORAGE.TRANS] handle_tx_commit_timeout (ob_trans_service_v4.cpp:393) [861656][T1_TransTimeWhe][T1][Y0-0000000000000000-0-0] [lt=116] handle tx commit timeout(ret=0, tx_id={txid:169795}, ref_cnt=4, cb_executed=false) [2024-04-24 10:00:49.581088] INFO [STORAGE.TRANS] handle_tx_commit_result_ (ob_trans_service_v4.cpp:549) [861400][pnio1][T1][YB420A000003-000616C2ACD902CF-0-0] [lt=103] handle tx commit result(ret=0, ref_cnt_0=4, tx={this:0x14b186574fe0, tx_id:{txid:169795}, state:5, addr:"10.0.0.3:2882", tenant_id:1, session_id:3221540160, assoc_session_id:3221540160, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924049402283, op_sn:4, alloc_ts:1713924049403545, active_ts:1713924049403545, commit_ts:1713924049409760, finish_ts:-1, timeout_us:86400000000, lock_timeout_us:-1, expire_ts:1714010449403545, coord_id:{id:1}, parts:[{id:{id:1}, addr:"10.0.0.2:2882", epoch:1430291097262351, first_scn:1713924049406085, last_scn:1713924049406085, last_touch_ts:1}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:2, commit_cb:0x14b1187d4988, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:1713924059410543, commit_task_.is_registered():true, ref:4}, commit_fin=false, result=-4023) [2024-04-24 10:00:49.581250] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2366) [861400][pnio1][T1][YB420A000003-000616C2ACD902CF-0-0] [lt=147][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:169795}, sender_ls_id={id:9223372036854775807}, receiver_ls_id={id:1}, msg_type=20, status=-4023, receiver_addr="10.0.0.2:2882", request_id=4) [2024-04-24 10:00:49.581344] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:285) [861400][pnio1][T0][YB420A000003-000616C2ACD902CF-0-0] [lt=84][errcode=0] trx rpc callback(ret=0, status=-4023, dst="10.0.0.2:2882", result={status:-4023, send_timestamp:1713924049578126, private_data:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:49.604922] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=310] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.605025] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=85] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.605097] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=66] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.613435] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [861686][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=127] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-24 10:00:49.619927] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [874532][T1004_LSArchive][T1004][YB420A000003-000616C2B568F0E7-0-0] [lt=88] gc stale ls task succ [2024-04-24 10:00:49.629370] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=91][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.630715] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=71] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.639506] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=67] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-04-24 10:00:49.645208] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [861685][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=114] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=7, size_used=15955, mem_used=27036672) [2024-04-24 10:00:49.647168] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=99] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.651451] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [861673][T1_LSMetaCh][T1][YB420A000003-000616C2A909ADA3-0-0] [lt=151] fetch root ls success(local_cluster_id=1713874979, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.2:2882", sql_port:2881, role:1, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:5, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713921656566585, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.3:2882", sql_port:2881, role:2, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["10.0.0.2:2882", "10.0.0.3:2882"]) [2024-04-24 10:00:49.655056] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=149] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.659348] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [861687][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=85] ====== checkpoint timer task ====== [2024-04-24 10:00:49.661609] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [861687][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=120] succ to get rec scn(*this={this:0x14b185a04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1713895202277178000, v:0}, last_push_gc_task_ts_:1713895204788534, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:950}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-24 10:00:49.661754] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [861687][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=138] tx ctx memtable get rec scn(this={ObITable:{this:0x14b186be0080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1713895204792049, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x14b186be0080, snapshot_version:{val:1713895204792049, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-04-24 10:00:49.661868] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [861687][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=105] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-04-24 10:00:49.662176] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [861687][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=118] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1713895261489897001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1713895261489897001, v:0}}, }, *this={this:0x14b186b16510, is_inited:true, freezing_scn:{val:1713895202277178000, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895204295552002, v:0}, clog_base_lsn:{lsn:67104768}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1713875366171467000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924058636985000, latest_log_ts:{val:1713924041985593001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713918035834957001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1713903554859671002, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:2, proposal_id:0, palf_env_:0x14b1945f8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:49.662366] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [861687][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=165] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x14b186b161b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x14b186b1bcd0, table_type:1, memtable_head:4, memtable_tail:5, t3m:0x14b1b09c4030, tables:[null, null, null, null, 0x14b186bbc080, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:1713895207645424, tx_data_table:0x14b186b1d1d0, ls_tablet_svr:0x14b186b16190, slice_allocator:0x14b186b1d210}, min_rec_scn={val:1713895204295552002, v:0}, ls_id_={id:1}) [2024-04-24 10:00:49.662494] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [861687][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=99] get rec log scn(service_type_=0, rec_log_ts={val:1713924041985593001, v:0}) [2024-04-24 10:00:49.662600] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [861687][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=97] get rec log scn(service_type_=1, rec_log_ts={val:1713918035834957001, v:0}) [2024-04-24 10:00:49.662691] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [861687][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=79] get rec log scn(service_type_=2, rec_log_ts={val:1713903554859671002, v:0}) [2024-04-24 10:00:49.662775] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [861687][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=72] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713895204295552002, v:0}, checkpoint_scn_in_ls_meta={val:1713895204295552002, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-04-24 10:00:49.662928] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [861687][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=136][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, curr_base_lsn={lsn:67104768}, new_base_lsn={lsn:67104768}, lsn={lsn:67104768}) [2024-04-24 10:00:49.663007] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [861687][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=69] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:67104768}, ls->get_ls_id()={id:1}) [2024-04-24 10:00:49.663095] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [861687][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=82] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-04-24 10:00:49.664764] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=62][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.669518] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [861691][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=91] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-04-24 10:00:49.669692] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [861691][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=87] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=9775) [2024-04-24 10:00:49.677575] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=89] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.677658] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=75] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.693349] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [874738][T1003_Flush][T1003][Y0-0000000000000000-0-0] [lt=101] ====== traversal_flush timer task ====== [2024-04-24 10:00:49.694122] INFO [MDS] flush (mds_table_mgr.cpp:195) [874738][T1003_Flush][T1003][Y0-0000000000000000-0-0] [lt=70] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1713895315527235001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1713895315527235003, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1713895315792183001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1713895315808716001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1713895315902633002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1713895315940096001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1713895316011635002, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1713895316147622001, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1713895316235730001, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1713895316235730002, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1713895316235730003, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1713895316390571001, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1713895316423756001, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1713895316423756002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1713895316591099001, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1713895316841701002, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1713895318470989001, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1713895318885339001, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1713895318943029001, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1713895319132417001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1713895319132417002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1713895319132417004, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1713895319340844003, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1713895319906507001, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1713895320985207001, v:0}}, {tablet_id:{id:60342}, rec_scn:{val:1713895320985207002, v:0}}, {tablet_id:{id:378}, rec_scn:{val:1713895320985207003, v:0}}, {tablet_id:{id:101036}, rec_scn:{val:1713895321186148001, v:0}}, {tablet_id:{id:60005}, rec_scn:{val:1713895321265127001, v:0}}, {tablet_id:{id:60130}, rec_scn:{val:1713895324336311004, v:0}}, {tablet_id:{id:101064}, rec_scn:{val:1713895324431298001, v:0}}, {tablet_id:{id:404}, rec_scn:{val:1713895324506362002, v:0}}, }, max_consequent_callbacked_scn={val:1713924048824341000, v:0}, *this={this:0x14b0f0a70510, is_inited:true, freezing_scn:{val:1713878494144158000, v:0}, ls:{ls_meta:{tenant_id:1003, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895201246660000, v:0}, clog_base_lsn:{lsn:67104768}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1713877340363486001, v:0}, all_id_meta:{id_meta:[{limited_id:1713924059270392000, latest_log_ts:{val:1713924042654251001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713900544098309001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1713877439572285001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:7, palf_env_:0x14b14219e030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:49.694408] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [874738][T1003_Flush][T1003][Y0-0000000000000000-0-0] [lt=268] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-04-24 10:00:49.698847] WDIAG [SQL.ENG] inner_get_next_row (ob_px_dist_transmit_op.cpp:73) [861725][T1_L5_G0][T1][YB420A000002-000616C2ACBCD98A-0-0] [lt=104][errcode=-4008] fail to exec ObPxTransmitOp::inner_get_next_row()(ret=-4008) [2024-04-24 10:00:49.705066] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=53] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.705207] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=124] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.705297] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=82] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.707727] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [874706][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=166] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:769, free_obj_hold_cnt:74, allocator used:3351768, allocator total:3466624}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:72, free_obj_hold_cnt:32, allocator used:6821464, allocator total:6821464}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():154752, full allocator used:1600, full allocator total:220288}, tablets_mem=10508376, tablets_mem_limit=257698020, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:23, free_obj_hold_cnt:163, allocator used:369024, allocator total:392448}, pending_cnt=0, wait_gc_count=0, tablet count=841) [2024-04-24 10:00:49.715312] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=92] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8890687488, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:49.716120] WDIAG [SQL.ENG] inner_get_next_row (ob_px_dist_transmit_op.cpp:73) [861725][T1_L5_G0][T1][YB420A000002-000616C2ACBCD98A-0-0] [lt=150][errcode=-4008] fail to exec ObPxTransmitOp::inner_get_next_row()(ret=-4008) [2024-04-24 10:00:49.719261] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [861520][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=170] new mysql sessid created(conn.sessid_=3221542346, support_ssl=false) [2024-04-24 10:00:49.719449] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [861520][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=93] sm conn init succ(conn.sessid_=3221542346, sess.client_addr_="127.0.0.1:48572") [2024-04-24 10:00:49.719564] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [861520][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=102] accept one succ(*s={this:0x14b11d060030, session_id:3221542346, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:168}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:49.719985] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [861520][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=189] username and tenantname(user_name=root, tenant_name=) [2024-04-24 10:00:49.720109] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [861520][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=91] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-04-24 10:00:49.720552] INFO load_privilege_info (obmp_connect.cpp:532) [861676][T1_MysqlQueueTh][T1][Y0-000616C2A9892E65-0-0] [lt=78] no tenant name set, use default tenant name(tenant_name=sys) [2024-04-24 10:00:49.721314] WDIAG [COMMON] get (ob_link_hashmap.h:313) [861676][T1_MysqlQueueTh][T1][Y0-000616C2A9892E65-0-0] [lt=200][errcode=0] uref leak check(node->uref_=1073788695, lbt()="0x121f36ac 0x5547c4b 0xac68123 0xac676b8 0xac3f33a 0xa3599e4 0xa3534fa 0x4e21201 0x13857852 0x13858aaa 0x13859029 0x12d23b01 0x12d1fdbc 0x14b1c1d132de 0x14b1c1a44e83") [2024-04-24 10:00:49.721520] INFO [SERVER] process (obmp_connect.cpp:475) [861676][T1_MysqlQueueTh][T1][Y0-000616C2A9892E65-0-0] [lt=76] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=%, sessid=3221542346, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-04-24 10:00:49.729519] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=137][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.730920] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=126] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.736817] INFO [SQL.JO] compute_table_location (ob_join_order.cpp:278) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1E9D-0-0] [lt=81] succeed to calculate base table sharding info(table_id=18446744073709551614, ref_table_id=104, is_global_index=false) [2024-04-24 10:00:49.736971] INFO [SQL.JO] init_est_sel_info_for_access_path (ob_join_order.cpp:12189) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1E9D-0-0] [lt=125] total rowcount, use statistics(table_meta_info_.table_row_count_=10, table_meta_info_.average_row_size_=2.060000000000000000e+02, table_meta_info_.micro_block_count_=1, table_meta_info_.part_size_=2.060000000000000000e+03) [2024-04-24 10:00:49.738452] INFO do_work (ob_rl_mgr.cpp:709) [861413][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=78] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-24 10:00:49.747361] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=104] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.750594] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [862263][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=86] ====== check clog disk timer task ====== [2024-04-24 10:00:49.750689] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [862263][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=84] get_disk_usage(ret=0, capacity(MB):=5529, used(MB):=1369) [2024-04-24 10:00:49.751179] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [862276][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=119] start do ls ha handler(ls_id_array_=[{id:1}, {id:1002}, {id:1001}]) [2024-04-24 10:00:49.755247] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=76] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.756080] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2630-0-0] [lt=103][errcode=-4023] tx is 2pc logging(this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:49.756271] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2630-0-0] [lt=175][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:49.756470] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2630-0-0] [lt=192][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338713}) [2024-04-24 10:00:49.756570] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2630-0-0] [lt=92][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924049751402, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.756683] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2630-0-0] [lt=66][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924049751402, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.764956] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=163][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.776563] INFO [CLOG] run1 (ob_garbage_collector.cpp:1338) [874508][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=66] Garbage Collector is running(seq_=4663, gc_interval=10000000) [2024-04-24 10:00:49.776786] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1473) [874508][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=132] self is leader, skip it(ls->get_ls_id()={id:1}) [2024-04-24 10:00:49.776953] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1473) [874508][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=161] self is leader, skip it(ls->get_ls_id()={id:1001}) [2024-04-24 10:00:49.777179] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1E9D-0-0] [lt=80][errcode=0] set logical del time(cache_obj->get_logical_del_time()=576362351292, cache_obj->added_lc()=false, cache_obj->get_object_id()=39598, cache_obj->get_tenant_id()=1, lbt()="0x121f36ac 0xc327dd4 0x4e3fdbe 0x4e31e28 0x4e29b67 0x4e21201 0x4e1eb17 0x4e0e175 0xa7ac354 0x12d1fdbc 0x14b1c1d132de 0x14b1c1a44e83") [2024-04-24 10:00:49.777834] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=77] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.777980] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=137] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.778619] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:272) [874508][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=112] GC check ls in member list, skip it(id={id:1002}, leader="10.0.0.2:2882") [2024-04-24 10:00:49.778689] INFO [STORAGE.TRANS] handle_tx_commit_timeout (ob_trans_service_v4.cpp:393) [861656][T1_TransTimeWhe][T1][Y0-0000000000000000-0-0] [lt=115] handle tx commit timeout(ret=0, tx_id={txid:169795}, ref_cnt=4, cb_executed=false) [2024-04-24 10:00:49.778733] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1431) [874508][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=97] gc_check_member_list_ cost time(ret=0, time_us=2016) [2024-04-24 10:00:49.778901] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [874508][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=96] execute_gc cost time(ret=0, time_us=13) [2024-04-24 10:00:49.780225] INFO [STORAGE.TRANS] handle_tx_commit_result_ (ob_trans_service_v4.cpp:549) [861402][pnio1][T1][YB420A000003-000616C2ACD902D0-0-0] [lt=91] handle tx commit result(ret=0, ref_cnt_0=4, tx={this:0x14b186574fe0, tx_id:{txid:169795}, state:5, addr:"10.0.0.3:2882", tenant_id:1, session_id:3221540160, assoc_session_id:3221540160, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924049402283, op_sn:4, alloc_ts:1713924049403545, active_ts:1713924049403545, commit_ts:1713924049409760, finish_ts:-1, timeout_us:86400000000, lock_timeout_us:-1, expire_ts:1714010449403545, coord_id:{id:1}, parts:[{id:{id:1}, addr:"10.0.0.2:2882", epoch:1430291097262351, first_scn:1713924049406085, last_scn:1713924049406085, last_touch_ts:1}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:3, commit_cb:0x14b1187d4988, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:1713924059410543, commit_task_.is_registered():true, ref:4}, commit_fin=false, result=-4023) [2024-04-24 10:00:49.780369] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2366) [861402][pnio1][T1][YB420A000003-000616C2ACD902D0-0-0] [lt=138][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:169795}, sender_ls_id={id:9223372036854775807}, receiver_ls_id={id:1}, msg_type=20, status=-4023, receiver_addr="10.0.0.2:2882", request_id=4) [2024-04-24 10:00:49.780429] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:285) [861402][pnio1][T0][YB420A000003-000616C2ACD902D0-0-0] [lt=54][errcode=0] trx rpc callback(ret=0, status=-4023, dst="10.0.0.2:2882", result={status:-4023, send_timestamp:1713924049778509, private_data:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:49.780600] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1626) [874508][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=56] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2024-04-24 10:00:49.782194] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1626) [874508][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=98] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}) [2024-04-24 10:00:49.782490] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=113] new mysql sessid created(conn.sessid_=3221542244, support_ssl=false) [2024-04-24 10:00:49.782631] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=69] sm conn init succ(conn.sessid_=3221542244, sess.client_addr_="10.0.0.3:48132") [2024-04-24 10:00:49.782763] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=107] accept one succ(*s={this:0x14b11d040030, session_id:3221542244, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:49.782991] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:943) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=152][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x14b11d040030, session_id:3221542244, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:49.783102] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=101] kill and revert session(conn.sessid_=3221542244, proxy_sessid=0, server_id=1, ret=0) [2024-04-24 10:00:49.783214] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:918) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=105] sock ref clean, do destroy(*s={this:0x14b11d040030, session_id:3221542244, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:49.783374] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:223) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=152] mark session id unused(conn.sessid_=3221542244) [2024-04-24 10:00:49.783462] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=80] connection close(sessid=3221542244, proxy_sessid=0, tenant_id=0, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:0], disconnect_state=0) [2024-04-24 10:00:49.783708] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [874796][T1003_L0_G5][T1003][YB420A000002-000616C2BEDDD122-0-0] [lt=113] get number(ret=0, service_type_=0, range=1, base_id=1713924049783522000, start_id=1713924049783522000, end_id=1713924049783522001) [2024-04-24 10:00:49.784037] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1626) [874508][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=95] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}) [2024-04-24 10:00:49.784198] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1648) [874508][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=116] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}]) [2024-04-24 10:00:49.784335] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1648) [874508][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=110] ls status is normal, skip(id={id:1001}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}]) [2024-04-24 10:00:49.784445] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1648) [874508][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=76] ls status is normal, skip(id={id:1002}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}]) [2024-04-24 10:00:49.784543] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [874508][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=76] execute_gc cost time(ret=0, time_us=361) [2024-04-24 10:00:49.784617] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [874508][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=58] ObSafeDestroyHandler start process [2024-04-24 10:00:49.797409] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1961) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=89] dump_dag_status(dag_cnt=0, map_size=0) [2024-04-24 10:00:49.797512] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1971) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=96] dump_dag_status(running_dag_net_map_size=0, blocking_dag_net_list_size=0) [2024-04-24 10:00:49.797606] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=88] dump_dag_status(priority="PRIO_COMPACTION_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:49.797714] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=94] dump_dag_status(priority="PRIO_HA_HIGH", low_limit=8, up_limit=8, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:49.797824] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=94] dump_dag_status(priority="PRIO_COMPACTION_MID", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:49.797933] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=101] dump_dag_status(priority="PRIO_HA_MID", low_limit=5, up_limit=5, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:49.797998] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=59] dump_dag_status(priority="PRIO_COMPACTION_LOW", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:49.798073] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=68] dump_dag_status(priority="PRIO_HA_LOW", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:49.798146] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=67] dump_dag_status(priority="PRIO_DDL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:49.798203] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=50] dump_dag_status(priority="PRIO_DDL_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:49.798304] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=90] dump_dag_status(priority="PRIO_TTL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:49.798364] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=51] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:49.798444] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:49.798557] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=70] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:49.798636] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=73] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:49.798723] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=78] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:49.798793] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=59] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:49.798942] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=142] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:49.799012] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=64] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:49.799092] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:49.799169] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=60] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:49.799253] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=76] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:49.799335] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=73] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:49.799450] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=107] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, dag_count=0) [2024-04-24 10:00:49.799528] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=70] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-04-24 10:00:49.799624] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=83] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0) [2024-04-24 10:00:49.799776] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=142] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-04-24 10:00:49.799938] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=153] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0) [2024-04-24 10:00:49.800022] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=69] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-04-24 10:00:49.800108] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=80] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0) [2024-04-24 10:00:49.800214] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=99] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-04-24 10:00:49.800288] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=67] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:49.800383] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=89] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:49.800463] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=68] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:49.800537] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=67] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:49.800616] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=71] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:49.800772] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=149] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:49.800949] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=170] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:49.801058] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=84] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:49.801134] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=66] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:49.801207] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=66] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:49.801263] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=50] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:49.801370] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=93] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:49.801442] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=66] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:49.801512] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=61] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:49.801612] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=92] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:49.801691] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=72] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:49.801785] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=86] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:49.801862] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=69] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:49.802043] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=170] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:49.802118] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=68] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:49.802202] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:49.802266] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=58] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:49.802335] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:49.802422] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=79] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:49.802505] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:49.802571] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=57] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:49.802630] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=53] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:49.802724] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=86] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:49.802805] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=72] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:49.802911] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=58] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:49.803039] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=121] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-04-24 10:00:49.803119] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-04-24 10:00:49.803195] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=68] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-04-24 10:00:49.803267] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=66] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-04-24 10:00:49.803330] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=57] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:49.803392] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=55] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:49.803455] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=57] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:49.803524] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=63] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:49.803584] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=54] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:49.803654] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=60] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:49.803735] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=64] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:49.803797] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=54] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:49.803944] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=133] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:49.804125] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=169] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:49.804193] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:49.804283] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=83] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:49.804363] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=71] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:49.804456] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:49.804542] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=59] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:49.804611] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:49.804695] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=77] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:49.804753] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=52] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:49.804843] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:49.805010] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=157] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:49.805126] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=109] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:49.805198] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=60] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:49.805280] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=69] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:49.805316] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=119] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.805358] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=71] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:49.805419] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=67] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.805456] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=90] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:49.805494] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=66] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.805525] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=60] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:49.805634] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=100] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:49.805739] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=98] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:49.805829] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=81] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:49.805955] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=116] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:49.806041] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=76] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:49.806185] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=137] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:49.806270] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=77] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:49.806352] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=73] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:49.806408] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=49] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:49.806473] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=57] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:49.806547] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=68] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:49.806611] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=58] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:49.806682] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=64] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0) [2024-04-24 10:00:49.806737] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=49] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, scheduled_task_count=0) [2024-04-24 10:00:49.806815] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=71] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0) [2024-04-24 10:00:49.806929] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=106] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, scheduled_task_count=0) [2024-04-24 10:00:49.807014] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=77] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0) [2024-04-24 10:00:49.807070] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=50] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-04-24 10:00:49.807338] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=262] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0) [2024-04-24 10:00:49.807439] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=91] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-04-24 10:00:49.807513] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=64] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0) [2024-04-24 10:00:49.807582] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, scheduled_task_count=0) [2024-04-24 10:00:49.807696] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=104] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_net_count=0) [2024-04-24 10:00:49.807777] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=71] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_net_count=0) [2024-04-24 10:00:49.807919] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=74] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_net_count=0) [2024-04-24 10:00:49.807996] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=68] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_net_count=0) [2024-04-24 10:00:49.808086] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=67] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_net_count=0) [2024-04-24 10:00:49.808179] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=86] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_net_count=0) [2024-04-24 10:00:49.808302] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=115] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_net_count=0) [2024-04-24 10:00:49.808398] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=83] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_net_count=0) [2024-04-24 10:00:49.808482] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1991) [862223][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=76] dump_dag_status(total_worker_cnt=43, total_running_task_cnt=0, work_thread_num=43, scheduled_task_cnt=0) [2024-04-24 10:00:49.812571] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [862264][T1002_ObTimer][T1002][Y0-0000000000000000-0-0] [lt=67] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-04-24 10:00:49.812767] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [862264][T1002_ObTimer][T1002][Y0-0000000000000000-0-0] [lt=164] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-04-24 10:00:49.812831] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [862264][T1002_ObTimer][T1002][Y0-0000000000000000-0-0] [lt=59] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2024-04-24 10:00:49.812951] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [862264][T1002_ObTimer][T1002][Y0-0000000000000000-0-0] [lt=113] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2024-04-24 10:00:49.813026] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [862264][T1002_ObTimer][T1002][Y0-0000000000000000-0-0] [lt=61] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=3, times=9686) [2024-04-24 10:00:49.829749] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=80][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.831036] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=132] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.832547] WDIAG [STORAGE.TRANS] inner_lock_for_read (ob_tx_data_functor.cpp:290) [2325187][T1004_L0_G0][T1004][YB420A000002-000616C2B75D354C-0-0] [lt=75][errcode=-6004] lock_for_read need retry(ret=-6004, tx_data={tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048272353001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}, lock_for_read_arg_={mvcc_acc_ctx_:{type:1, abs_lock_timeout:1713924051532753, tx_lock_timeout:-1, snapshot:{version:{val:1713924049416995000, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x14b0efafe490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, data_trans_id_:{txid:176888}, data_sql_sequence_:1713924048659397, read_latest_:false, scn_:{val:1713924048664122000, v:0}}, tx_cc_ctx={state:40, prepare_version:{val:1713924048664122000, v:0}}) [2024-04-24 10:00:49.838549] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [861692][T1_LSArchiveMgr][T1][YB420A000003-000616C2A888F0E7-0-0] [lt=76] gc stale ls task succ [2024-04-24 10:00:49.839932] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=82] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-04-24 10:00:49.843248] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [874542][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=88] start do ls ha handler(ls_id_array_=[{id:1}, {id:1002}, {id:1001}]) [2024-04-24 10:00:49.847521] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=127] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.854746] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1073) [874714][T1003_ReplayPro][T1003][Y0-0000000000000000-0-0] [lt=73] replay status is not follower(min_unreplayed_lsn={lsn:283486487}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713921656870314, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:283486487}, next_to_submit_scn_:{val:1713900515126283548, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x14b12a08d000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:283486487}, end_lsn:{lsn:283486487}, read_buf:{buf_len_:8192, buf_:0x14b12a08d000}, block_size:67104768, log_storage_:0x14b0f0aa7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-04-24 10:00:49.854953] INFO [CLOG] operator() (ob_log_replay_service.cpp:1590) [874714][T1003_ReplayPro][T1003][Y0-0000000000000000-0-0] [lt=198] get_replay_process success(id={id:1}, replayed_log_size=283486487, unreplayed_log_size=0) [2024-04-24 10:00:49.855079] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:159) [874714][T1003_ReplayPro][T1003][Y0-0000000000000000-0-0] [lt=74] dump tenant replay process(tenant_id=1003, unreplayed_log_size(MB)=0, estimate_time(second)=0, replayed_log_size(MB)=270, last_replayed_log_size(MB)=270, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2024-04-24 10:00:49.855519] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=76] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.860837] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2320564][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B4-0-0] [lt=59][errcode=-4023] tx is 2pc logging(this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:49.861099] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2320564][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B4-0-0] [lt=253][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:49.861208] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2320564][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B4-0-0] [lt=103][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176913}) [2024-04-24 10:00:49.861274] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2320564][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B4-0-0] [lt=57][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924049856278, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.861383] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2320564][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B4-0-0] [lt=87][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924049856278, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.861451] INFO [STORAGE.TRANS] retransmit_downstream_msg_ (ob_two_phase_upstream_committer.cpp:311) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=85] unresponded participant(i=1, *this={this:0x14b1481c0010, ref:8, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048664959], ctx_create_time:1713924045155303}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79589464}, {lsn:79618713}], redo_log_no:2, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:6, data_.length():1313}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048664122000, v:0}, trans_type:2, next_log_entry_no:4, max_applied_log_ts:{val:1713924048664122000, v:0}, max_applying_log_ts:{val:1713924048664122000, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:1713924048659397, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:79620195}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048272353001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877443458344 max_table_version=1713877467338312 trans_version={val:1713924048664122000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1 callback_alloc_count=134 callback_free_count=0 checksum=141527899 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=158 redo_sync_succ_count=158 redo_sync_fail_count=0 main_list_length=158 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=158, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924048272353001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048665345, block_frozen_memtable:null}) [2024-04-24 10:00:49.861764] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [874436][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=84] ====== tenant freeze timer task ====== [2024-04-24 10:00:49.861812] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=285] rpc statistics(total_trans_msg_count=12, total_batch_msg_count=11) [2024-04-24 10:00:49.862016] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=130] handle timeout(ret=0, *this={this:0x14b1481c0010, ref:9, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048664959], ctx_create_time:1713924045155303}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79589464}, {lsn:79618713}], redo_log_no:2, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:6, data_.length():1313}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048664122000, v:0}, trans_type:2, next_log_entry_no:4, max_applied_log_ts:{val:1713924048664122000, v:0}, max_applying_log_ts:{val:1713924048664122000, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:1713924048659397, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:79620195}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048272353001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877443458344 max_table_version=1713877467338312 trans_version={val:1713924048664122000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1 callback_alloc_count=134 callback_free_count=0 checksum=141527899 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=158 redo_sync_succ_count=158 redo_sync_fail_count=0 main_list_length=158 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=158, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924048272353001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048665345, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=400000) [2024-04-24 10:00:49.862197] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=164] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:49.862472] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=78] handle timeout(ret=0, *this={this:0x14b055e1e050, ref:4, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924048589222}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88120890}], redo_log_no:1, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048658150001, v:0}, trans_type:2, next_log_entry_no:3, max_applied_log_ts:{val:1713924048658150001, v:0}, max_applying_log_ts:{val:1713924048658150001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:88121631}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048588043001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=0 trans_version={val:1713924048658150001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=1 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924048588043001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048589222, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=400000) [2024-04-24 10:00:49.862667] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=160] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:49.862988] INFO [STORAGE.TRANS] handle_2pc_commit_request_impl_ (ob_two_phase_downstream_committer.cpp:420) [2319281][T1004_L0_G0][T1004][Y0-0000000000000000-0-0] [lt=59] committer is under logging(ret=0, *this={this:0x14b055e1e050, ref:4, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924048589222}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88120890}], redo_log_no:1, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048658150001, v:0}, trans_type:2, next_log_entry_no:3, max_applied_log_ts:{val:1713924048658150001, v:0}, max_applying_log_ts:{val:1713924048658150001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:88121631}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048588043001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=0 trans_version={val:1713924048658150001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=1 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924048588043001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048589222, block_frozen_memtable:null}) [2024-04-24 10:00:49.864137] INFO [STORAGE.TRANS] handle_2pc_commit_request_impl_ (ob_two_phase_downstream_committer.cpp:420) [2319281][T1004_L0_G0][T1004][Y0-0000000000000000-0-0] [lt=144] committer is under logging(ret=0, *this={this:0x14b055e1e050, ref:4, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924048589222}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88120890}], redo_log_no:1, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048658150001, v:0}, trans_type:2, next_log_entry_no:3, max_applied_log_ts:{val:1713924048658150001, v:0}, max_applying_log_ts:{val:1713924048658150001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:88121631}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048588043001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=0 trans_version={val:1713924048658150001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=1 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924048588043001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048589222, block_frozen_memtable:null}) [2024-04-24 10:00:49.865113] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=88][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.867808] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:426) [874701][T1003_FreInfoRe][T1003][Y0-0000000000000000-0-0] [lt=63] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2024-04-24 10:00:49.868116] INFO [STORAGE] update_info (ob_tenant_freeze_info_mgr.cpp:766) [874701][T1003_FreInfoRe][T1003][Y0-0000000000000000-0-0] [lt=173] ObTenantFreezeInfoMgr success to update infos(snapshot_gc_ts=1713924041269740000, min_major_snapshot=1, cur_idx=0, info_list_[0]=[{freeze_version:1, schema_version:1, cluster_version:17180000768}, {freeze_version:1713895200575122000, schema_version:1713878493426008, cluster_version:17180000768}], info_list_[1]=[{freeze_version:1, schema_version:1, cluster_version:17180000768}, {freeze_version:1713895200575122000, schema_version:1713878493426008, cluster_version:17180000768}], snapshots_[0]=[], snapshots_[1]=[]) [2024-04-24 10:00:49.869559] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [861689][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=65] ====== check clog disk timer task ====== [2024-04-24 10:00:49.869680] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [861689][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=107] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=235) [2024-04-24 10:00:49.869918] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [861689][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=140] cannot_recycle_log_size statistics(cannot_recycle_log_size=112643346, threshold=644245094, need_update_checkpoint_scn=false) [2024-04-24 10:00:49.878001] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=123] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.878126] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=112] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.887309] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1457) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2B9DC616A-0-0] [lt=153] receive register_req from acceptor, ignore(self="10.0.0.3:2882", palf_id=1, child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924049881080, keepalive_ts:-1}) [2024-04-24 10:00:49.887423] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1476) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2B9DC616A-0-0] [lt=103] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="10.0.0.3:2882", child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924049881080, keepalive_ts:-1}, is_to_leader=true) [2024-04-24 10:00:49.896645] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [861461][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=111] skip inspect bad block(last_check_time=1713875167993818, last_macro_idx=-1) [2024-04-24 10:00:49.905492] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=61] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.905625] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=97] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.905736] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=103] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.915845] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [862342][T1001_PlanCache][T1001][Y0-0000000000000000-0-0] [lt=71] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2024-04-24 10:00:49.916045] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [862342][T1001_PlanCache][T1001][Y0-0000000000000000-0-0] [lt=175] start lib cache evict(tenant_id=1001, mem_hold=48234496, mem_limit=53687090, cache_obj_num=184, cache_node_num=184) [2024-04-24 10:00:49.916144] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=114] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8894881792, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:49.916155] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [862342][T1001_PlanCache][T1001][Y0-0000000000000000-0-0] [lt=99] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=184, cache_node_num=184) [2024-04-24 10:00:49.916240] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [862342][T1001_PlanCache][T1001][Y0-0000000000000000-0-0] [lt=75] schedule next cache evict task(evict_interval=5000000) [2024-04-24 10:00:49.918028] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=96] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-24 10:00:49.918231] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=132] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=286, get_gts_with_stc_cnt=25, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=23, try_wait_gts_elapse_cnt=0) [2024-04-24 10:00:49.918383] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=126] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1713924049918167], gts:1713924049917601000, latest_srr:[mts=1713924049918167]}) [2024-04-24 10:00:49.918566] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [861494][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=83] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=610) [2024-04-24 10:00:49.918735] INFO [LIB] stat (utility.h:1197) [861494][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=144] [PALF STAT APPEND COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=0, this=0x14b147074e50, extra_info="id:1") [2024-04-24 10:00:49.918833] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:157) [861494][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=72] submit log success(service_type_=0, last_id=1713924066491232000, limited_id=1713924066491232000) [2024-04-24 10:00:49.918986] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [861494][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=145] submit log with lock success(service_type_=0, last_id=1713924066491232000, limited_id=1713924066491232000) [2024-04-24 10:00:49.919107] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=59] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1713924049918474], gts:1713924049917601000, latest_srr:[mts=1713924049918474]}) [2024-04-24 10:00:49.920054] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [862342][T1001_PlanCache][T1001][Y0-0000000000000000-0-0] [lt=70] Dumping All Cache Objs(alloc_obj_list.count()=184, alloc_obj_list=[{obj_id:432, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:1887, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188985}, {obj_id:1938, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178961}, {obj_id:1942, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191942}, {obj_id:5690, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200417}, {obj_id:6208, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179593}, {obj_id:6639, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196868}, {obj_id:6864, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:4, added_to_lc:true, mem_used:191152}, {obj_id:8127, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188048}, {obj_id:8376, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194797}, {obj_id:8862, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186066}, {obj_id:8910, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174048}, {obj_id:9123, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172112}, {obj_id:9128, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177974}, {obj_id:9200, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174912}, {obj_id:9224, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190249}, {obj_id:9230, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184112}, {obj_id:9256, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172528}, {obj_id:9257, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175256}, {obj_id:9259, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217216}, {obj_id:9261, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179235}, {obj_id:9268, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181217}, {obj_id:9280, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175314}, {obj_id:9282, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189174}, {obj_id:9300, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9302, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9307, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181421}, {obj_id:9308, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191482}, {obj_id:9311, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178816}, {obj_id:9312, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185947}, {obj_id:9315, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185840}, {obj_id:9319, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193660}, {obj_id:9320, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186978}, {obj_id:9321, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200494}, {obj_id:9324, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194633}, {obj_id:9325, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194633}, {obj_id:9330, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189428}, {obj_id:9351, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217516}, {obj_id:9352, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215079}, {obj_id:9359, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:267194}, {obj_id:9411, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197500}, {obj_id:9412, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185946}, {obj_id:9413, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:205036}, {obj_id:9422, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9424, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175259}, {obj_id:9425, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176132}, {obj_id:9426, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183681}, {obj_id:9427, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184692}, {obj_id:9428, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186090}, {obj_id:9429, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195616}, {obj_id:9430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175846}, {obj_id:9431, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9432, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9433, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9434, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176254}, {obj_id:9435, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9436, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202640}, {obj_id:9437, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:276579}, {obj_id:9438, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193474}, {obj_id:9439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204472}, {obj_id:9440, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9441, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9442, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188297}, {obj_id:9443, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178961}, {obj_id:9444, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175106}, {obj_id:9445, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175786}, {obj_id:9446, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182028}, {obj_id:9447, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173408}, {obj_id:9448, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198112}, {obj_id:9449, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173104}, {obj_id:9450, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177587}, {obj_id:9451, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193848}, {obj_id:9452, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179880}, {obj_id:9453, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187481}, {obj_id:9454, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174463}, {obj_id:9455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181217}, {obj_id:9456, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184692}, {obj_id:9457, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179603}, {obj_id:9458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9459, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173504}, {obj_id:9460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9461, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177273}, {obj_id:9462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9463, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237328}, {obj_id:9464, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183952}, {obj_id:9465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216328}, {obj_id:9466, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172736}, {obj_id:9467, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172688}, {obj_id:9468, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174304}, {obj_id:9469, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9470, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:171872}, {obj_id:9471, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:171872}, {obj_id:9472, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180401}, {obj_id:9473, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239808}, {obj_id:9474, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173748}, {obj_id:9475, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176522}, {obj_id:9476, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186515}, {obj_id:9477, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179903}, {obj_id:9478, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183760}, {obj_id:9479, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187845}, {obj_id:9480, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9481, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184351}, {obj_id:9482, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187634}, {obj_id:9483, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172960}, {obj_id:9484, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173648}, {obj_id:9485, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194627}, {obj_id:9486, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199487}, {obj_id:9487, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174080}, {obj_id:9488, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176993}, {obj_id:9489, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190852}, {obj_id:9490, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188653}, {obj_id:9491, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179176}, {obj_id:9492, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175554}, {obj_id:9493, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184112}, {obj_id:9494, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191482}, {obj_id:9495, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179880}, {obj_id:9496, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182028}, {obj_id:9497, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178561}, {obj_id:9498, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185202}, {obj_id:9499, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174304}, {obj_id:9500, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183270}, {obj_id:9501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175217}, {obj_id:9502, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9503, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9505, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9506, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189479}, {obj_id:9507, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184648}, {obj_id:9508, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190740}, {obj_id:9509, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174320}, {obj_id:9510, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191972}, {obj_id:9511, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196560}, {obj_id:9512, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186258}, {obj_id:9513, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186090}, {obj_id:9514, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188876}, {obj_id:9515, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192134}, {obj_id:9516, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198118}, {obj_id:9517, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188489}, {obj_id:9518, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194036}, {obj_id:9519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188552}, {obj_id:9520, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176132}, {obj_id:9521, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175072}, {obj_id:9522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9523, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9525, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177840}, {obj_id:9526, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179045}, {obj_id:9527, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9528, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230954}, {obj_id:9529, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230563}, {obj_id:9530, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224241}, {obj_id:9531, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185060}, {obj_id:9532, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185060}, {obj_id:9533, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210481}, {obj_id:9534, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9535, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9536, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9537, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207393}, {obj_id:9538, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177480}, {obj_id:9539, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177520}, {obj_id:9540, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:255944}, {obj_id:9541, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9542, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185840}, {obj_id:9543, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187248}, {obj_id:9544, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:231809}, {obj_id:9545, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:227196}, {obj_id:9546, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180682}, {obj_id:9547, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236301}, {obj_id:9548, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234010}, {obj_id:9549, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173232}, {obj_id:9550, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173808}, {obj_id:9551, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233644}, {obj_id:9552, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220312}, {obj_id:9553, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:282552}, {obj_id:9554, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177480}, {obj_id:9555, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177520}, {obj_id:9556, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179903}, {obj_id:9557, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184839}, {obj_id:9558, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187642}, {obj_id:9559, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194259}, {obj_id:9560, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186988}, {obj_id:9561, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230281}, {obj_id:9562, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9563, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174048}]) [2024-04-24 10:00:49.920630] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [862342][T1001_PlanCache][T1001][Y0-0000000000000000-0-0] [lt=549] schedule next cache evict task(evict_interval=5000000) [2024-04-24 10:00:49.929977] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=111][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.931274] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=127] tx gc loop thread is running(MTL_ID()=1001) [2024-04-24 10:00:49.931355] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=72] try gc retain ctx [2024-04-24 10:00:49.932583] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=153] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895203696037000, v:0}, clog_base_lsn:{lsn:134209536}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1713875407740689002, v:0}, all_id_meta:{id_meta:[{limited_id:1713924059338456000, latest_log_ts:{val:1713924042694950001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713898575276749001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1713910854505533001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:5, palf_env_:0x14b157ff2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-04-24 10:00:49.932787] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=164] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.942207] INFO eloop_run (eloop.c:143) [861404][pnio2][T0][Y0-0000000000000000-0-0] [lt=109] PNIO [ratelimit] time: 1713924049942199, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007573, add_bytes: 0 [2024-04-24 10:00:49.942725] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [862131][T1002_Occam][T1002][YB420A000003-000616C2AFD988F2-0-0] [lt=100] for each ls(succ_num=3, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:49.943002] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [862266][T1002_LSArchive][T1002][YB420A000003-000616C2AE58F0E7-0-0] [lt=189] gc stale ls task succ [2024-04-24 10:00:49.944249] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [862169][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=73] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2024-04-24 10:00:49.944363] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [862169][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=103] start lib cache evict(tenant_id=1002, mem_hold=48234496, mem_limit=53687090, cache_obj_num=183, cache_node_num=182) [2024-04-24 10:00:49.944456] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [862169][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=79] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=183, cache_node_num=182) [2024-04-24 10:00:49.944590] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [862169][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=116] schedule next cache evict task(evict_interval=5000000) [2024-04-24 10:00:49.947797] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=95] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.947819] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [862169][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=104] Dumping All Cache Objs(alloc_obj_list.count()=183, alloc_obj_list=[{obj_id:2721, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:2790, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174859}, {obj_id:4132, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189174}, {obj_id:4482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179325}, {obj_id:4520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172112}, {obj_id:5006, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172528}, {obj_id:5672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175256}, {obj_id:6010, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6012, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6718, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6750, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6751, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6752, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176850}, {obj_id:6764, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6769, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179498}, {obj_id:6793, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191778}, {obj_id:6794, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6795, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6796, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181286}, {obj_id:6800, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197346}, {obj_id:6802, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183681}, {obj_id:6812, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172688}, {obj_id:6813, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6816, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216328}, {obj_id:6817, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172080}, {obj_id:6818, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6819, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157847}, {obj_id:6820, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239808}, {obj_id:6821, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217216}, {obj_id:6822, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6823, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180970}, {obj_id:6824, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:6825, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186020}, {obj_id:6826, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175846}, {obj_id:6827, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173696}, {obj_id:6828, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172960}, {obj_id:6829, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173648}, {obj_id:6830, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173408}, {obj_id:6831, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175259}, {obj_id:6832, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175106}, {obj_id:6833, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175786}, {obj_id:6834, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181421}, {obj_id:6835, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183681}, {obj_id:6836, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173104}, {obj_id:6837, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177587}, {obj_id:6838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180401}, {obj_id:6839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179235}, {obj_id:6840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187845}, {obj_id:6841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187787}, {obj_id:6843, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182229}, {obj_id:6845, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193665}, {obj_id:6846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179603}, {obj_id:6850, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173504}, {obj_id:6851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177273}, {obj_id:6853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237328}, {obj_id:6855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200494}, {obj_id:6856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199487}, {obj_id:6857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180401}, {obj_id:6858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6859, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6860, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190158}, {obj_id:6861, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6862, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6863, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:6864, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6865, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193516}, {obj_id:6866, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6867, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175774}, {obj_id:6868, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6869, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173168}, {obj_id:6870, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173744}, {obj_id:6871, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174080}, {obj_id:6872, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176993}, {obj_id:6873, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183952}, {obj_id:6874, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183359}, {obj_id:6875, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211056}, {obj_id:6876, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6877, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6878, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6879, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6880, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:367564}, {obj_id:6881, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176958}, {obj_id:6882, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175416}, {obj_id:6883, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173104}, {obj_id:6884, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174640}, {obj_id:6885, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6886, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:305014}, {obj_id:6887, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6888, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6889, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6890, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189947}, {obj_id:6891, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6892, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6893, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6894, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178966}, {obj_id:6895, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:205860}, {obj_id:6896, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177765}, {obj_id:6897, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6898, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189922}, {obj_id:6899, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196325}, {obj_id:6900, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6901, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6902, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6903, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6904, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181108}, {obj_id:6905, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6906, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6907, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:6908, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174930}, {obj_id:6909, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175978}, {obj_id:6910, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181421}, {obj_id:6911, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191158}, {obj_id:6912, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191970}, {obj_id:6913, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6914, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179525}, {obj_id:6915, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6916, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6917, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6918, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6919, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180419}, {obj_id:6920, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6921, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6922, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6923, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6924, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6925, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6926, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6927, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:166010}, {obj_id:6928, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:162310}, {obj_id:6929, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172272}, {obj_id:6930, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179932}, {obj_id:6931, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182243}, {obj_id:6932, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181861}, {obj_id:6933, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6934, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179391}, {obj_id:6935, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188005}, {obj_id:6936, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6937, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6938, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194633}, {obj_id:6939, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194633}, {obj_id:6940, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183528}, {obj_id:6941, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183528}, {obj_id:6942, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6943, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6944, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:238004}, {obj_id:6945, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194209}, {obj_id:6946, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6947, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6948, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6949, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6950, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197398}, {obj_id:6951, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196679}, {obj_id:6952, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6953, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183031}, {obj_id:6954, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185836}, {obj_id:6955, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176038}, {obj_id:6956, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173888}, {obj_id:6957, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176365}, {obj_id:6958, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6959, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:171872}, {obj_id:6960, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:171872}, {obj_id:6961, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174320}, {obj_id:6962, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6963, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6964, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220368}, {obj_id:6965, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:287598}, {obj_id:6966, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:282792}, {obj_id:6967, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6968, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6969, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173104}, {obj_id:6970, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177587}, {obj_id:6971, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6972, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6973, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173408}, {obj_id:6974, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6975, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}]) [2024-04-24 10:00:49.948200] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [862169][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=362] schedule next cache evict task(evict_interval=5000000) [2024-04-24 10:00:49.955739] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=147] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:49.958269] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2631-0-0] [lt=62][errcode=-4023] tx is 2pc logging(this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:49.958480] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2631-0-0] [lt=201][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:49.958694] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2631-0-0] [lt=201][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338713}) [2024-04-24 10:00:49.958912] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2631-0-0] [lt=152][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924049953662, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.959006] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2631-0-0] [lt=78][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924049953662, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:49.965375] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=66][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:49.978197] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=92] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:49.978313] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=103] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:49.979189] INFO [STORAGE.TRANS] handle_tx_commit_timeout (ob_trans_service_v4.cpp:393) [861656][T1_TransTimeWhe][T1][Y0-0000000000000000-0-0] [lt=94] handle tx commit timeout(ret=0, tx_id={txid:169795}, ref_cnt=4, cb_executed=false) [2024-04-24 10:00:49.981240] INFO [STORAGE.TRANS] handle_tx_commit_result_ (ob_trans_service_v4.cpp:549) [861401][pnio1][T1][YB420A000003-000616C2ACD902D1-0-0] [lt=73] handle tx commit result(ret=0, ref_cnt_0=4, tx={this:0x14b186574fe0, tx_id:{txid:169795}, state:5, addr:"10.0.0.3:2882", tenant_id:1, session_id:3221540160, assoc_session_id:3221540160, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924049402283, op_sn:4, alloc_ts:1713924049403545, active_ts:1713924049403545, commit_ts:1713924049409760, finish_ts:-1, timeout_us:86400000000, lock_timeout_us:-1, expire_ts:1714010449403545, coord_id:{id:1}, parts:[{id:{id:1}, addr:"10.0.0.2:2882", epoch:1430291097262351, first_scn:1713924049406085, last_scn:1713924049406085, last_touch_ts:1}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:4, commit_cb:0x14b1187d4988, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:1713924059410543, commit_task_.is_registered():true, ref:4}, commit_fin=false, result=-4023) [2024-04-24 10:00:49.981391] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2366) [861401][pnio1][T1][YB420A000003-000616C2ACD902D1-0-0] [lt=145][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:169795}, sender_ls_id={id:9223372036854775807}, receiver_ls_id={id:1}, msg_type=20, status=-4023, receiver_addr="10.0.0.2:2882", request_id=4) [2024-04-24 10:00:49.981485] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:285) [861401][pnio1][T0][YB420A000003-000616C2ACD902D1-0-0] [lt=85][errcode=0] trx rpc callback(ret=0, status=-4023, dst="10.0.0.2:2882", result={status:-4023, send_timestamp:1713924049979034, private_data:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:49.995830] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2720) [861318][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=125][errcode=-4002] get invalid Ethernet speed, use default(devname="eth0") [2024-04-24 10:00:49.995972] WDIAG [SERVER] runTimerTask (ob_server.cpp:3239) [861318][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=133][errcode=-4002] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4002, ret="OB_INVALID_ARGUMENT") [2024-04-24 10:00:50.000340] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [862447][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=113] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:50.005620] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=80] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.005764] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=117] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.005849] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=77] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.014366] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [861686][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=109] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1713924048840067000, v:0}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1713924050014221}, version_delta=1174291) [2024-04-24 10:00:50.030203] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=114][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.031518] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=66] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.040290] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=111] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-04-24 10:00:50.047988] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=144] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.055789] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=93] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.061987] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B5-0-0] [lt=129][errcode=-4023] tx is 2pc logging(this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:50.062179] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B5-0-0] [lt=180][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:50.062318] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B5-0-0] [lt=131][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176913}) [2024-04-24 10:00:50.062388] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B5-0-0] [lt=61][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924050057370, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:50.062464] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B5-0-0] [lt=63][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924050057370, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:50.065545] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=85] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"10.0.0.3:2882", switch_to_leader_ts:-1}) [2024-04-24 10:00:50.065637] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=79] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"10.0.0.3:2882", switch_to_leader_ts:-1}) [2024-04-24 10:00:50.065831] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=185][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.070768] INFO [STORAGE] exec_wait (ob_tmp_file_cache.cpp:1531) [1556243][T1004_ComTimerT][T1004][Y0-0000000000000000-0-0] [lt=118] succeed to do one round of tmp block io(ret=-4023, loop_nums=0, wait_io_cnt=0, washing_count=0, block_cache_num=0, page_cache_num=0) [2024-04-24 10:00:50.078306] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=84] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.078397] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=82] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.081727] INFO eloop_run (eloop.c:143) [861399][pnio1][T0][Y0-0000000000000000-0-0] [lt=117] PNIO [ratelimit] time: 1713924050081721, bytes: 14676402985, bw: 0.107926 MB/s, add_ts: 1007693, add_bytes: 114039 [2024-04-24 10:00:50.105834] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=114] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.105959] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=112] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.106070] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=101] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.109226] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [862405][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=110] handle timeout(ret=0, *this={this:0x14b15a16b090, ref:4, trans_id:{txid:568574}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924078927990, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048929680], ctx_create_time:1713924048928190}{ls_id:{id:1}, session_id:3221542296, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.3:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048917318, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048929552000, v:0}, log_ts:{val:1713924048929552000, v:0}, lsn:{lsn:314133761}, submit_ts:1713924048929552}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:568574}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:568574}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048929552000, v:0}, start_scn:{val:1713924048929552000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875429046312 max_table_version=1713875429046312 trans_version={val:1713924048929552000, v:0} commit_version={val:1713924048929552000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:568574} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=593250361 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048929552000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048929552, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=100000) [2024-04-24 10:00:50.109419] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [862405][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=181] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1713924048929552000, v:0}, log_ts:{val:1713924048929552000, v:0}, lsn:{lsn:314133761}, submit_ts:1713924048929552}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:568574}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924048929552000, v:0}, log_ts:{val:1713924048929552000, v:0}, lsn:{lsn:314133761}, submit_ts:1713924048929552}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:568574}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:50.116997] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [874706][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=128] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:769, free_obj_hold_cnt:74, allocator used:3351768, allocator total:3466624}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:72, free_obj_hold_cnt:32, allocator used:6821464, allocator total:6821464}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():154752, full allocator used:1600, full allocator total:220288}, tablets_mem=10508376, tablets_mem_limit=257698020, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:23, free_obj_hold_cnt:163, allocator used:369024, allocator total:392448}, pending_cnt=0, wait_gc_count=0, tablet count=841) [2024-04-24 10:00:50.117065] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=74] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8894881792, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:50.130355] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=63][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.130794] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [862406][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=168] handle timeout(ret=0, *this={this:0x14b15a1754d0, ref:4, trans_id:{txid:568573}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924078910775, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048911462], ctx_create_time:1713924048910937}{ls_id:{id:1}, session_id:3221542304, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.3:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909715, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048910937000, v:0}, log_ts:{val:1713924048910937000, v:0}, lsn:{lsn:314133219}, submit_ts:1713924048910937}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568573}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:568573}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048910937000, v:0}, start_scn:{val:1713924048910937000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875429046312 max_table_version=1713875429046312 trans_version={val:1713924048910937000, v:0} commit_version={val:1713924048910937000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:568573} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=2854444523 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048910937000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048910937, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=100000) [2024-04-24 10:00:50.131091] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [862406][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=291] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1713924048910937000, v:0}, log_ts:{val:1713924048910937000, v:0}, lsn:{lsn:314133219}, submit_ts:1713924048910937}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568573}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924048910937000, v:0}, log_ts:{val:1713924048910937000, v:0}, lsn:{lsn:314133219}, submit_ts:1713924048910937}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568573}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:50.131766] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=96] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.141993] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [874742][T1003_LSArchive][T1003][YB420A000003-000616C2B608F0E7-0-0] [lt=91] gc stale ls task succ [2024-04-24 10:00:50.148142] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=77] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.156025] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=135] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.165731] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=142][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.165973] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [862309][T1001_Occam][T1001][YB420A000003-000616C2AFF988F2-0-0] [lt=84] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:50.178482] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=57] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.178626] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=135] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.196577] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [862434][T1001_CKClogDis][T1001][Y0-0000000000000000-0-0] [lt=81] ====== check clog disk timer task ====== [2024-04-24 10:00:50.196707] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [862434][T1001_CKClogDis][T1001][Y0-0000000000000000-0-0] [lt=118] get_disk_usage(ret=0, capacity(MB):=614, used(MB):=363) [2024-04-24 10:00:50.196824] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [862434][T1001_CKClogDis][T1001][Y0-0000000000000000-0-0] [lt=71] cannot_recycle_log_size statistics(cannot_recycle_log_size=179923627, threshold=193273528, need_update_checkpoint_scn=false) [2024-04-24 10:00:50.206015] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=64] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.206141] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=110] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.206283] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=129] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.220332] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861495][RpcIO][T0][Y0-0000000000000000-0-0] [lt=148] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:50.220415] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861496][RpcIO][T0][Y0-0000000000000000-0-0] [lt=134] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:50.220950] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861498][RpcIO][T0][Y0-0000000000000000-0-0] [lt=72] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:50.221228] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861497][RpcIO][T0][Y0-0000000000000000-0-0] [lt=117] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:50.221408] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [874754][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=101] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:50.230514] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=81][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.231966] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=166] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.240753] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=112] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-04-24 10:00:50.248325] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=79] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.248447] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [874722][T1003_LockWaitM][T1003][Y0-0000000000000000-0-0] [lt=92] report RowHolderMapper summary info(count=4, bkt_cnt=248) [2024-04-24 10:00:50.256285] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=83] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.259152] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2632-0-0] [lt=66][errcode=-4023] tx is 2pc logging(this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:50.259423] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2632-0-0] [lt=263][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:50.259637] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2632-0-0] [lt=207][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338713}) [2024-04-24 10:00:50.259774] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2632-0-0] [lt=126][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924050254474, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:50.259996] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2632-0-0] [lt=196][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924050254474, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:50.261453] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [874525][T1004_ReqMemEvi][T1004][Y0-0000000000000000-0-0] [lt=140] sql audit evict task end(request_manager_->get_tenant_id()=1004, evict_high_mem_level=278313880, evict_high_size_level=90000, evict_batch_count=0, elapse_time=7, size_used=80585, mem_used=131023872) [2024-04-24 10:00:50.262672] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B6-0-0] [lt=79][errcode=-4023] tx is 2pc logging(this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:50.262857] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B6-0-0] [lt=139][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:50.263016] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B6-0-0] [lt=151][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176913}) [2024-04-24 10:00:50.263075] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B6-0-0] [lt=49][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924050257688, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:50.263184] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B6-0-0] [lt=85][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924050257688, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:50.265916] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=84][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.269410] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [862437][T1001_LSArchive][T1001][YB420A000003-000616C2AEF8F0E7-0-0] [lt=129] gc stale ls task succ [2024-04-24 10:00:50.278674] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=132] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.278768] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=78] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.279950] INFO [STORAGE.TRANS] handle_tx_commit_timeout (ob_trans_service_v4.cpp:393) [861656][T1_TransTimeWhe][T1][Y0-0000000000000000-0-0] [lt=112] handle tx commit timeout(ret=0, tx_id={txid:169795}, ref_cnt=4, cb_executed=false) [2024-04-24 10:00:50.281784] INFO [STORAGE.TRANS] handle_tx_commit_result_ (ob_trans_service_v4.cpp:549) [861400][pnio1][T1][YB420A000003-000616C2ACD902D2-0-0] [lt=65] handle tx commit result(ret=0, ref_cnt_0=4, tx={this:0x14b186574fe0, tx_id:{txid:169795}, state:5, addr:"10.0.0.3:2882", tenant_id:1, session_id:3221540160, assoc_session_id:3221540160, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924049402283, op_sn:4, alloc_ts:1713924049403545, active_ts:1713924049403545, commit_ts:1713924049409760, finish_ts:-1, timeout_us:86400000000, lock_timeout_us:-1, expire_ts:1714010449403545, coord_id:{id:1}, parts:[{id:{id:1}, addr:"10.0.0.2:2882", epoch:1430291097262351, first_scn:1713924049406085, last_scn:1713924049406085, last_touch_ts:1}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:5, commit_cb:0x14b1187d4988, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:1713924059410543, commit_task_.is_registered():true, ref:4}, commit_fin=false, result=-4023) [2024-04-24 10:00:50.281957] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2366) [861400][pnio1][T1][YB420A000003-000616C2ACD902D2-0-0] [lt=164][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:169795}, sender_ls_id={id:9223372036854775807}, receiver_ls_id={id:1}, msg_type=20, status=-4023, receiver_addr="10.0.0.2:2882", request_id=4) [2024-04-24 10:00:50.282042] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:285) [861400][pnio1][T0][YB420A000003-000616C2ACD902D2-0-0] [lt=78][errcode=0] trx rpc callback(ret=0, status=-4023, dst="10.0.0.2:2882", result={status:-4023, send_timestamp:1713924050279730, private_data:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:50.284346] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [862534][T1002_L0_G5][T1002][YB420A000002-000616C2BEDDD125-0-0] [lt=117] is logging [2024-04-24 10:00:50.284461] WDIAG [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:116) [862534][T1002_L0_G5][T1002][YB420A000002-000616C2BEDDD125-0-0] [lt=80][errcode=-4023] submit log callback use too mush time(submit_log_ts=1713924049918819, cb={log_ts:{val:1713924049917601001, v:0}, log_buf:"", pos:40, id_srv_type:0, limited_id:1713924066491232000}, service_type=0, latest_log_ts={val:1713924039886464001, v:0}, self="10.0.0.3:2882") [2024-04-24 10:00:50.284556] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [862534][T1002_L0_G5][T1002][YB420A000002-000616C2BEDDD125-0-0] [lt=88] submit log with lock(ret=-4023, service_type_=0, last_id=1713924066491232000, limited_id=1713924066491232000) [2024-04-24 10:00:50.299097] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:83) [862256][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=90] XA scheduler heartbeat task statistics(avg_time=7) [2024-04-24 10:00:50.306189] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=104] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.306365] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=113] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.306483] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=106] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.318063] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=94] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8894881792, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:50.330959] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=160][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.332797] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=137] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.336704] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [874739][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=127] ====== check clog disk timer task ====== [2024-04-24 10:00:50.336914] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [874739][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=142] get_disk_usage(ret=0, capacity(MB):=3686, used(MB):=348) [2024-04-24 10:00:50.337078] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [874739][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=90] cannot_recycle_log_size statistics(cannot_recycle_log_size=231417561, threshold=1159641169, need_update_checkpoint_scn=false) [2024-04-24 10:00:50.348491] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=93] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.349528] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [874735][T1003_ReqMemEvi][T1003][Y0-0000000000000000-0-0] [lt=96] sql audit evict task end(request_manager_->get_tenant_id()=1003, evict_high_mem_level=19327352, evict_high_size_level=90000, evict_batch_count=0, elapse_time=8, size_used=8146, mem_used=14558208) [2024-04-24 10:00:50.356509] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=101] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.366056] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=102][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.367893] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=107] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: MICRO_BLOCK_CACHE_IO, mode: read, size: 4096.00, iops: 23.85, rt: 659.79) [2024-04-24 10:00:50.368031] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=83] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:50.368561] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=126] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 948.00) [2024-04-24 10:00:50.368777] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=173] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:50.370692] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=82] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-24 10:00:50.370859] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=150] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-24 10:00:50.372760] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:834) [862160][T1002_BalanceEx][T1002][YB420A000003-000616C2B3D9ADA1-0-0] [lt=52] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2024-04-24 10:00:50.372982] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:171) [862160][T1002_BalanceEx][T1002][YB420A000003-000616C2B3D9ADA1-0-0] [lt=212] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2024-04-24 10:00:50.378824] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=133] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.378945] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=115] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.406381] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=74] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.406491] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=85] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.406569] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=72] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.410208] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [862405][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=108] handle timeout(ret=0, *this={this:0x14b15a16b090, ref:4, trans_id:{txid:568574}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924078927990, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048929680], ctx_create_time:1713924048928190}{ls_id:{id:1}, session_id:3221542296, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.3:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048917318, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048929552000, v:0}, log_ts:{val:1713924048929552000, v:0}, lsn:{lsn:314133761}, submit_ts:1713924048929552}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:568574}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:568574}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048929552000, v:0}, start_scn:{val:1713924048929552000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875429046312 max_table_version=1713875429046312 trans_version={val:1713924048929552000, v:0} commit_version={val:1713924048929552000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:568574} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=593250361 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048929552000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048929552, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=200000) [2024-04-24 10:00:50.410464] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [862405][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=239] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1713924048929552000, v:0}, log_ts:{val:1713924048929552000, v:0}, lsn:{lsn:314133761}, submit_ts:1713924048929552}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:568574}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924048929552000, v:0}, log_ts:{val:1713924048929552000, v:0}, lsn:{lsn:314133761}, submit_ts:1713924048929552}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:568574}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:50.411250] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [861593][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=105] ====== tenant freeze timer task ====== [2024-04-24 10:00:50.417497] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [861702][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=92] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:50.420272] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [861494][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=84] is logging [2024-04-24 10:00:50.420389] WDIAG [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:116) [861494][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=79][errcode=-4023] submit log callback use too mush time(submit_log_ts=1713924049918819, cb={log_ts:{val:1713924049917601001, v:0}, log_buf:"", pos:40, id_srv_type:0, limited_id:1713924066491232000}, service_type=0, latest_log_ts={val:1713924039886464001, v:0}, self="10.0.0.3:2882") [2024-04-24 10:00:50.420493] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [861494][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=79] submit log with lock(ret=-4023, service_type_=0, last_id=1713924066491232000, limited_id=1713924066491232000) [2024-04-24 10:00:50.421081] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=91] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1713924050421004], gts:1713924050420739000, latest_srr:[mts=1713924050421004]}) [2024-04-24 10:00:50.431639] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=137][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.432240] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [862406][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=75] handle timeout(ret=0, *this={this:0x14b15a1754d0, ref:4, trans_id:{txid:568573}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924078910775, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048911462], ctx_create_time:1713924048910937}{ls_id:{id:1}, session_id:3221542304, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.3:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909715, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048910937000, v:0}, log_ts:{val:1713924048910937000, v:0}, lsn:{lsn:314133219}, submit_ts:1713924048910937}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568573}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:568573}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048910937000, v:0}, start_scn:{val:1713924048910937000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875429046312 max_table_version=1713875429046312 trans_version={val:1713924048910937000, v:0} commit_version={val:1713924048910937000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:568573} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=2854444523 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048910937000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048910937, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=200000) [2024-04-24 10:00:50.432524] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [862406][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=260] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1713924048910937000, v:0}, log_ts:{val:1713924048910937000, v:0}, lsn:{lsn:314133219}, submit_ts:1713924048910937}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568573}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924048910937000, v:0}, log_ts:{val:1713924048910937000, v:0}, lsn:{lsn:314133219}, submit_ts:1713924048910937}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568573}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:50.433024] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=162] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.435747] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2648) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=113] [PALF_DUMP](palf_id=1001, self="10.0.0.3:2882", [SlidingWindow]={palf_id:1001, self:"10.0.0.3:2882", lsn_allocator:{max_log_id:673424, max_lsn:1002197017, max_scn:{val:1713924048914830000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:1002196895}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:960253977}}, last_submit_lsn:{lsn:1002196895}, last_submit_end_lsn:{lsn:1002197017}, last_submit_log_id:673424, last_submit_log_pid:7, max_flushed_lsn:{lsn:1002195322}, max_flushed_end_lsn:{lsn:1002196895}, max_flushed_log_pid:7, committed_end_lsn:{lsn:1002196895}, last_slide_log_id:673423, last_slide_scn:{val:1713924048836268000, v:0}, last_slide_lsn:{lsn:1002195322}, last_slide_end_lsn:{lsn:1002196895}, last_slide_log_pid:7, last_slide_log_accum_checksum:2368686946, last_fetch_end_lsn:{lsn:993740449}, last_fetch_max_log_id:640060, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1713921654976913, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x14b1336ea530}, [StateMgr]={this:0x14b1336ee2b0, self:"10.0.0.3:2882", palf_id:1001, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}, leader:"10.0.0.3:2882", leader_epoch:10, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:951799104}, scan_disk_log_finished:true, last_check_start_id:673424, is_changing_config_with_arb:false, reconfirm_start_time_us:1713921655879811, palf_role_change_cb:0x14b1336f4b70, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"10.0.0.3:2882", alive_paxos_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, alive_paxos_replica_num:2, log_ms_meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:951798982}, prev_mode_pid:7}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:6, prev_lsn:{lsn:951798982}, prev_end_lsn:{lsn:951799104}, prev_mode_pid:7}, reconfig_barrier:{prev_log_proposal_id:6, prev_lsn:{lsn:951798982}, prev_end_lsn:{lsn:951799104}, prev_mode_pid:7}, persistent_config_version:{proposal_id:7, config_seq:8}, ms_ack_list:0, resend_config_version:{proposal_id:7, config_seq:8}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x14b1336ebc70}, [ModeMgr]={palf_id:1001, self:"10.0.0.3:2882", applied_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1713875572267468000, v:0}}, accepted_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1713875572267468000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1713875572267468000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1713921656484602, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1713875572267468001, v:0}, base_lsn_for_block_gc:{lsn:335523840}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}, log_config_meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:951798982}, prev_mode_pid:7}, log_snapshot_meta:{version:1, base_lsn:{lsn:335523840}, prev_log_info:{log_id:-1, lsn:{lsn:18446744073709551615}, scn:{val:18446744073709551615, v:3}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1713875572267468000, v:0}}}, log_meta_storage:{log_tail:{lsn:110592}, readable_log_tail:{lsn:110592}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1006571520}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/data/admin/test/oceanbase/store/clog/tenant_1002/1001/meta", dir_fd:158, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66994176, block_header_serialize_buf_:0x14b1336f0438, flashback_version:0}, log_storage:{log_tail:{lsn:1002196895}, readable_log_tail:{lsn:1002196895}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:939466752}, min_scn:{val:1713921045787116001, v:0}, curr_block_id:14, palf_id:1001}, block_mgr:{log_dir:"/data/admin/test/oceanbase/store/clog/tenant_1002/1001/log", dir_fd:163, min_block_id:0, max_block_id:15, curr_writable_block_id:14}, logical_block_size_:67104768, curr_block_writable_size_:4374625, block_header_serialize_buf_:0x14b1336f20b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1713921655880756, this:0x14b1336ee9f0}, [Reconfirm]={palf_id:1001, self:"10.0.0.3:2882", state:"FINISHED", new_proposal_id:7, prepare_log_ack_list:1"10.0.0.2:2882";, curr_paxos_follower_list:1{server:"10.0.0.2:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"10.0.0.3:2882", majority_max_accept_pid:6, majority_max_lsn:{lsn:951799104}, saved_end_lsn:{lsn:951799104}, last_submit_prepare_req_time_us:1713921655881991, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:1713921656581154, last_purge_throttling_time_us:-1, this:0x14b1336ee430}) [2024-04-24 10:00:50.436032] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2652) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=276] [PALF_DUMP](palf_id=1001, self="10.0.0.3:2882", ack_info_list=[{member:{server:"10.0.0.2:2882", timestamp:1, flag:0}, last_ack_time_us:1713924048883940, last_flushed_end_lsn:{lsn:1002196895}}, {member:{server:"10.0.0.3:2882", timestamp:1, flag:0}, last_ack_time_us:1713924048882614, last_flushed_end_lsn:{lsn:1002196895}}]) [2024-04-24 10:00:50.436131] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4290) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=85] current log_task status(palf_id=1001, self="10.0.0.3:2882", log_id=673424, log_task={header:{begin_lsn:{lsn:1002196895}, end_lsn:{lsn:1002197017}, log_id:673424, min_scn:{val:1713924048914830000, v:0}, max_scn:{val:1713924048914830000, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1002196895}, data_checksum:2222604892, accum_checksum:817730249, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1713924049103909, freeze_ts:1713924049103926, submit_ts:1713924049104061, flushed_ts:-1, gen_to_freeze cost time:17, gen_to_submit cost time:152, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"10.0.0.2:2882", timestamp:1, flag:0}, last_ack_time_us:1713924048883940, last_flushed_end_lsn:{lsn:1002196895}}, {member:{server:"10.0.0.3:2882", timestamp:1, flag:0}, last_ack_time_us:1713924048882614, last_flushed_end_lsn:{lsn:1002196895}}]) [2024-04-24 10:00:50.441116] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=157] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-04-24 10:00:50.445313] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861507][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=61] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:50.446104] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861508][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=101] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:50.446448] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861509][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=68] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:50.447856] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861511][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=86] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:50.448498] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861500][BatchIO][T0][Y0-0000000000000000-0-0] [lt=71] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:50.448767] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861502][BatchIO][T0][Y0-0000000000000000-0-0] [lt=92] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:50.448786] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861506][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=66] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:50.448820] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=107] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.448993] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861504][BatchIO][T0][Y0-0000000000000000-0-0] [lt=142] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:50.449153] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861501][BatchIO][T0][Y0-0000000000000000-0-0] [lt=76] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:50.449425] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861503][BatchIO][T0][Y0-0000000000000000-0-0] [lt=108] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:50.450665] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861513][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=113] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:50.455838] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [874648][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=82] ====== tenant freeze timer task ====== [2024-04-24 10:00:50.456913] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=87] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.466178] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=213][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.479059] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=88] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.479167] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=99] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.480204] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [861656][T1_TransTimeWhe][T1][Y0-0000000000000000-0-0] [lt=117] rpc statistics(total_trans_msg_count=5, total_batch_msg_count=0) [2024-04-24 10:00:50.480335] INFO [STORAGE.TRANS] handle_tx_commit_timeout (ob_trans_service_v4.cpp:393) [861656][T1_TransTimeWhe][T1][Y0-0000000000000000-0-0] [lt=96] handle tx commit timeout(ret=0, tx_id={txid:169795}, ref_cnt=4, cb_executed=false) [2024-04-24 10:00:50.482074] INFO [STORAGE.TRANS] handle_tx_commit_result_ (ob_trans_service_v4.cpp:549) [861401][pnio1][T1][YB420A000003-000616C2ACD902D3-0-0] [lt=79] handle tx commit result(ret=0, ref_cnt_0=4, tx={this:0x14b186574fe0, tx_id:{txid:169795}, state:5, addr:"10.0.0.3:2882", tenant_id:1, session_id:3221540160, assoc_session_id:3221540160, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924049402283, op_sn:4, alloc_ts:1713924049403545, active_ts:1713924049403545, commit_ts:1713924049409760, finish_ts:-1, timeout_us:86400000000, lock_timeout_us:-1, expire_ts:1714010449403545, coord_id:{id:1}, parts:[{id:{id:1}, addr:"10.0.0.2:2882", epoch:1430291097262351, first_scn:1713924049406085, last_scn:1713924049406085, last_touch_ts:1}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:6, commit_cb:0x14b1187d4988, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:1713924059410543, commit_task_.is_registered():true, ref:4}, commit_fin=false, result=-4023) [2024-04-24 10:00:50.482220] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2366) [861401][pnio1][T1][YB420A000003-000616C2ACD902D3-0-0] [lt=140][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:169795}, sender_ls_id={id:9223372036854775807}, receiver_ls_id={id:1}, msg_type=20, status=-4023, receiver_addr="10.0.0.2:2882", request_id=4) [2024-04-24 10:00:50.482324] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:285) [861401][pnio1][T0][YB420A000003-000616C2ACD902D3-0-0] [lt=85][errcode=0] trx rpc callback(ret=0, status=-4023, dst="10.0.0.2:2882", result={status:-4023, send_timestamp:1713924050480090, private_data:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:50.491354] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [862147][T1002_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=110] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):5529, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):5529, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1049, sequence:1}) [2024-04-24 10:00:50.506474] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=158] tx gc loop thread is running(MTL_ID()=1002) [2024-04-24 10:00:50.506613] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=113] try gc retain ctx [2024-04-24 10:00:50.508246] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=157] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895310268028001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924044057788000, v:0}, tablet_change_checkpoint_scn:{val:1713875507773116000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924056491232000, latest_log_ts:{val:1713924039886464001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713890395635598001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:5, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-04-24 10:00:50.508451] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=167] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.509814] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=74] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1713895385940970001, v:0}, clog_base_lsn:{lsn:335523840}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924044057788000, v:0}, tablet_change_checkpoint_scn:{val:1713890381649962000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:7, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-04-24 10:00:50.510023] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=175] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.511371] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=86] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_create_status:1, clog_checkpoint_scn:{val:1713895298278308001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924044057788000, v:0}, tablet_change_checkpoint_scn:{val:1713890384892147000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:2, proposal_id:0, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:1942234752, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-04-24 10:00:50.511537] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=100] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.516610] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [862171][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=74] ====== tenant freeze timer task ====== [2024-04-24 10:00:50.516795] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [862171][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=118] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=75497472, ctx.total_memstore_used_=109051904, ctx.total_memstore_hold_=109051904, memstore_freeze_trigger_limit_=91749580, tenant_id=1002) [2024-04-24 10:00:50.516985] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [862171][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=162] [TenantFreezer] oldest frozen memtable(list="") [2024-04-24 10:00:50.518938] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=81] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8894881792, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:50.520952] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=78] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=0, clean_num=31457) [2024-04-24 10:00:50.522326] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [861898][T1_L0_G0][T1][YB420A000002-000616C2AA6C607C-0-0] [lt=83] [RS_MGR] force set rs list(master_rs="10.0.0.2:2882") [2024-04-24 10:00:50.522458] INFO [SERVER] handle_heartbeat (ob_service.cpp:3094) [861898][T1_L0_G0][T1][YB420A000002-000616C2AA6C607C-0-0] [lt=107] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"10.0.0.3:2882", server_id:1, rs_addr:"10.0.0.2:2882", rs_server_status:1, epoch_id:5}, hb_response={zone:"zone1", server:"10.0.0.3:2882", sql_port:2881, build_version:"4.2.2.0_100010012024022719-c984fe7cb7a4cef85a40323a0d073f0c9b7b8235(Feb 27 2024 19:21:00)", start_service_time:1713875297913711, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=138) [2024-04-24 10:00:50.522464] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:834) [874425][T1004_BalanceEx][T1004][YB420A000003-000616C2BB69A618-0-0] [lt=122] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2024-04-24 10:00:50.522552] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:171) [874425][T1004_BalanceEx][T1004][YB420A000003-000616C2BB69A618-0-0] [lt=79] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2024-04-24 10:00:50.522745] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [874706][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=133] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:769, free_obj_hold_cnt:74, allocator used:3351768, allocator total:3466624}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:72, free_obj_hold_cnt:32, allocator used:6821464, allocator total:6821464}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():154752, full allocator used:1600, full allocator total:220288}, tablets_mem=10508376, tablets_mem_limit=257698020, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:23, free_obj_hold_cnt:163, allocator used:369024, allocator total:392448}, pending_cnt=0, wait_gc_count=0, tablet count=841) [2024-04-24 10:00:50.523504] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [861558][T1_Occam][T1][YB420A000003-000616C2A92989A6-0-0] [lt=106] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:50.531801] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=122][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.533200] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=89] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.549054] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=173] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.557074] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=93] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.559700] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [874764][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2633-0-0] [lt=140][errcode=-4023] tx is 2pc logging(this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:50.559924] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [874764][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2633-0-0] [lt=214][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:50.560083] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [874764][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2633-0-0] [lt=147][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338713}) [2024-04-24 10:00:50.560151] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [874764][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2633-0-0] [lt=62][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924050555043, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:50.560225] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [874764][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2633-0-0] [lt=61][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924050555043, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:50.562978] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B7-0-0] [lt=107][errcode=-4023] tx is 2pc logging(this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:50.563138] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B7-0-0] [lt=153][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:50.563277] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B7-0-0] [lt=125][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176913}) [2024-04-24 10:00:50.563372] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B7-0-0] [lt=86][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924050558302, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:50.563483] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B7-0-0] [lt=93][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924050558302, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:50.563994] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [862259][T1002_ReqMemEvi][T1002][Y0-0000000000000000-0-0] [lt=68] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=16106127, evict_high_size_level=90000, evict_batch_count=0, elapse_time=9, size_used=6658, mem_used=12478464) [2024-04-24 10:00:50.566345] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=80][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.575435] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [862430][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=105] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=90000, evict_batch_count=0, elapse_time=12, size_used=8520, mem_used=14558208) [2024-04-24 10:00:50.579323] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=82] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.579445] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=88] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.598945] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [861744][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=82] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1713924044295272, tenant_id:1004, sequence_id:5497558139157}) [2024-04-24 10:00:50.599059] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [861744][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=103] try reload schema success(schema_version=1713878795542168, refresh_schema_info={schema_version:1713924044295272, tenant_id:1004, sequence_id:5497558139157}, schema_ret=0) [2024-04-24 10:00:50.599092] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EE-0-0] [lt=97] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1713924044295272, tenant_id:1004, sequence_id:5497558139157}) [2024-04-24 10:00:50.599249] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:280) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EE-0-0] [lt=95] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1713924044295272, tenant_id:1004, sequence_id:5497558139157}, schema_info={schema_version:1713924044295272, tenant_id:1004, sequence_id:5497558139157}) [2024-04-24 10:00:50.606804] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=77] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.606995] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=170] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.607081] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=76] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.615227] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [861686][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=104] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-24 10:00:50.620213] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [874532][T1004_LSArchive][T1004][YB420A000003-000616C2B568F0E7-0-0] [lt=104] gc stale ls task succ [2024-04-24 10:00:50.632038] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=117][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.633333] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=110] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.641415] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=67] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-04-24 10:00:50.645596] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [861685][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=117] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=8, size_used=15968, mem_used=27036672) [2024-04-24 10:00:50.649233] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=98] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.657203] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [874394][T1004_Occam][T1004][YB420A000003-000616C2B6C982E8-0-0] [lt=72] for each ls(succ_num=3, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:50.657282] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=89] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.664180] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [861673][T1_LSMetaCh][T1][YB420A000003-000616C2A909ADA4-0-0] [lt=280] fetch root ls success(local_cluster_id=1713874979, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.2:2882", sql_port:2881, role:1, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:5, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713921656566585, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.3:2882", sql_port:2881, role:2, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["10.0.0.2:2882", "10.0.0.3:2882"]) [2024-04-24 10:00:50.666475] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=88][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.667699] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1961) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=60] dump_dag_status(dag_cnt=0, map_size=0) [2024-04-24 10:00:50.667780] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1971) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=70] dump_dag_status(running_dag_net_map_size=0, blocking_dag_net_list_size=0) [2024-04-24 10:00:50.667844] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=58] dump_dag_status(priority="PRIO_COMPACTION_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:50.667960] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=108] dump_dag_status(priority="PRIO_HA_HIGH", low_limit=8, up_limit=8, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:50.668028] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=56] dump_dag_status(priority="PRIO_COMPACTION_MID", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:50.668102] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=68] dump_dag_status(priority="PRIO_HA_MID", low_limit=5, up_limit=5, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:50.668164] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=54] dump_dag_status(priority="PRIO_COMPACTION_LOW", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:50.668221] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=50] dump_dag_status(priority="PRIO_HA_LOW", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:50.668352] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=120] dump_dag_status(priority="PRIO_DDL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:50.668437] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=72] dump_dag_status(priority="PRIO_DDL_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:50.668512] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=67] dump_dag_status(priority="PRIO_TTL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:50.668603] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=85] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:50.668671] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=61] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:50.668750] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=72] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:50.668843] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=78] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:50.668953] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=99] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:50.669028] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=65] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:50.669101] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=67] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:50.669166] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=56] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:50.669226] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=55] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:50.669330] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=96] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:50.669415] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=77] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:50.669494] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=71] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:50.669557] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=56] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, dag_count=0) [2024-04-24 10:00:50.669638] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=75] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-04-24 10:00:50.669723] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=67] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0) [2024-04-24 10:00:50.669806] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=76] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-04-24 10:00:50.669917] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=104] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0) [2024-04-24 10:00:50.670002] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=77] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-04-24 10:00:50.670067] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=57] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0) [2024-04-24 10:00:50.670171] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=87] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-04-24 10:00:50.670298] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=115] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:50.670407] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=102] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:50.670488] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=73] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:50.670562] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=65] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:50.670673] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=103] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:50.670783] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=95] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:50.670922] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=128] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:50.670996] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=67] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:50.671122] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=116] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:50.671221] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=91] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:50.671312] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=85] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:50.671433] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=113] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:50.671512] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=67] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:50.671611] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=94] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:50.671693] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=75] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:50.671823] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=122] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:50.671941] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=108] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:50.672046] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=93] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:50.672144] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=83] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:50.672217] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:50.672326] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=97] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:50.672423] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=88] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:50.672496] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=67] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:50.672565] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=63] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:50.672632] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:50.672735] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=97] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:50.672802] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=61] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:50.672902] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=61] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:50.672989] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=69] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:50.673064] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=63] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:50.673149] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=76] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-04-24 10:00:50.673254] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=97] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-04-24 10:00:50.673344] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=83] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-04-24 10:00:50.673416] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=67] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-04-24 10:00:50.673500] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=78] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:50.673573] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=63] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:50.673673] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=85] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:50.673745] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=56] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:50.673811] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=59] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:50.673956] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=129] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:50.674053] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=87] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:50.674154] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=95] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:50.674212] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=53] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:50.674292] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=67] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:50.674397] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=96] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:50.674524] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=109] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:50.674598] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=65] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:50.674697] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=84] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:50.674772] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=60] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:50.674912] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=132] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:50.674994] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:50.675167] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=60] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:50.675256] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=77] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:50.675378] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=114] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:50.675465] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=76] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:50.675589] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=115] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:50.675726] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=114] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:50.675812] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=80] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:50.676129] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=307] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:50.676211] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:50.676291] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=67] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:50.676373] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:50.676452] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=69] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:50.676510] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=52] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:50.676604] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=81] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:50.676675] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:50.676759] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=77] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:50.676849] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=76] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:50.676968] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=109] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:50.677040] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=56] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:50.677112] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:50.677178] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=60] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:50.677273] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=83] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0) [2024-04-24 10:00:50.677365] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=82] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, scheduled_task_count=0) [2024-04-24 10:00:50.677486] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=112] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0) [2024-04-24 10:00:50.677566] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=71] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, scheduled_task_count=0) [2024-04-24 10:00:50.677701] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=126] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0) [2024-04-24 10:00:50.677775] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=66] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-04-24 10:00:50.677845] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=60] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0) [2024-04-24 10:00:50.677957] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=106] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-04-24 10:00:50.678027] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0) [2024-04-24 10:00:50.678102] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=66] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, scheduled_task_count=0) [2024-04-24 10:00:50.678168] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=60] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_net_count=0) [2024-04-24 10:00:50.678264] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=84] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_net_count=0) [2024-04-24 10:00:50.678337] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=65] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_net_count=0) [2024-04-24 10:00:50.678417] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=71] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_net_count=0) [2024-04-24 10:00:50.678497] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=72] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_net_count=0) [2024-04-24 10:00:50.678584] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=78] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_net_count=0) [2024-04-24 10:00:50.678677] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=79] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_net_count=0) [2024-04-24 10:00:50.678764] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=80] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_net_count=0) [2024-04-24 10:00:50.678836] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1991) [862396][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=64] dump_dag_status(total_worker_cnt=43, total_running_task_cnt=0, work_thread_num=43, scheduled_task_cnt=0) [2024-04-24 10:00:50.679373] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=80] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.679460] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=75] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.680527] INFO [STORAGE.TRANS] handle_tx_commit_timeout (ob_trans_service_v4.cpp:393) [861656][T1_TransTimeWhe][T1][Y0-0000000000000000-0-0] [lt=97] handle tx commit timeout(ret=0, tx_id={txid:169795}, ref_cnt=4, cb_executed=false) [2024-04-24 10:00:50.688399] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:270) [874394][T1004_Occam][T1004][YB420A000003-000616C2B6C982EA-0-0] [lt=92][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::try_gc_mds_table_(oceanbase::storage::ObTablet &), (*this=|threshold=5.00ms|start at 10:00:50.682|488=8us|total=5.82ms) [2024-04-24 10:00:50.688604] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:270) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=89][errcode=0] cost too much time:mds_table_impl.ipp:virtual int oceanbase::storage::mds::MdsTableImpl, oceanbase::st..., (*this=|threshold=5.00ms|start at 10:00:50.682|632=5.70ms|total=5.95ms) [2024-04-24 10:00:50.689848] INFO [STORAGE.TRANS] handle_tx_commit_result_ (ob_trans_service_v4.cpp:549) [861402][pnio1][T1][YB420A000003-000616C2ACD902D4-0-0] [lt=72] handle tx commit result(ret=0, ref_cnt_0=4, tx={this:0x14b186574fe0, tx_id:{txid:169795}, state:5, addr:"10.0.0.3:2882", tenant_id:1, session_id:3221540160, assoc_session_id:3221540160, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924049402283, op_sn:4, alloc_ts:1713924049403545, active_ts:1713924049403545, commit_ts:1713924049409760, finish_ts:-1, timeout_us:86400000000, lock_timeout_us:-1, expire_ts:1714010449403545, coord_id:{id:1}, parts:[{id:{id:1}, addr:"10.0.0.2:2882", epoch:1430291097262351, first_scn:1713924049406085, last_scn:1713924049406085, last_touch_ts:1}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:7, commit_cb:0x14b1187d4988, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:1713924059410543, commit_task_.is_registered():true, ref:4}, commit_fin=false, result=-4023) [2024-04-24 10:00:50.690049] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2366) [861402][pnio1][T1][YB420A000003-000616C2ACD902D4-0-0] [lt=185][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:169795}, sender_ls_id={id:9223372036854775807}, receiver_ls_id={id:1}, msg_type=20, status=-4023, receiver_addr="10.0.0.2:2882", request_id=4) [2024-04-24 10:00:50.690128] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:285) [861402][pnio1][T0][YB420A000003-000616C2ACD902D4-0-0] [lt=69][errcode=0] trx rpc callback(ret=0, status=-4023, dst="10.0.0.2:2882", result={status:-4023, send_timestamp:1713924050680400, private_data:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:50.695382] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [874529][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=65] ====== check clog disk timer task ====== [2024-04-24 10:00:50.695471] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [874529][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=74] get_disk_usage(ret=0, capacity(MB):=33177, used(MB):=390) [2024-04-24 10:00:50.707069] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=87] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.707173] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=90] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.707279] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=99] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.711118] INFO [LIB] stat (utility.h:1197) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=86] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=311745, this=0x14b0c0f013b0, extra_info="palf_id:1002") [2024-04-24 10:00:50.711216] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=91][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:40358611}], scn_array=[{val:1713924048832745000, v:0}], curr_size=122, accum_size=1970, time_cost=1808523) [2024-04-24 10:00:50.711333] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=109] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=1970) [2024-04-24 10:00:50.711471] INFO [LIB] stat (utility.h:1214) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=101] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=393, this=0x14b176ffd3b0, extra_info="") [2024-04-24 10:00:50.711779] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=121] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=168, this=0x14b0c0f015e0, extra_info="palf_id:1002") [2024-04-24 10:00:50.713092] INFO [LIB] stat (utility.h:1197) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=269] [PALF STAT WRITE LOG COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=506874, this=0x14b1470b53b0, extra_info="palf_id:1") [2024-04-24 10:00:50.713379] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=275][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67159339}], scn_array=[{val:1713924048836268000, v:0}], curr_size=122, accum_size=488, time_cost=1809072) [2024-04-24 10:00:50.713443] INFO [PALF] inner_write_once_ (log_block_handler.cpp:411) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=185] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=62734239, this={dio_aligned_buf:{buf_write_offset:25, buf_padding_size:4071, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:8269037, truncate_used_ts:9487502}, log_block_size:67108864, dir_fd:163, io_fd:137}, aligned_buf_len=8192, aligned_buf="", aligned_block_offset=62730240, buf_len=122, total_write_size_=158063, total_write_size_after_dio_=593920, ob_pwrite_used_ts=6415224, count=106, each_pwrite_cost=60520) [2024-04-24 10:00:50.713487] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=102] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=488) [2024-04-24 10:00:50.713633] INFO [LIB] stat (utility.h:1197) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=138] [PALF STAT WRITE LOG COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=809173, this=0x14b1336f53b0, extra_info="palf_id:1001") [2024-04-24 10:00:50.713702] INFO [LIB] stat (utility.h:1197) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=126] [PALF STAT WRITE LOG COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=865273, this=0x14b186b513b0, extra_info="palf_id:1") [2024-04-24 10:00:50.713711] INFO [LIB] stat (utility.h:1214) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=146] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=168, this=0x14b1679f6530, extra_info="") [2024-04-24 10:00:50.713722] INFO [LIB] stat (utility.h:1197) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=231] [PALF STAT WRITE LOG COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=824401, this=0x14b0efb333b0, extra_info="palf_id:1") [2024-04-24 10:00:50.713731] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [874495][T1004_OB_SLOG][T1004][Y0-0000000000000000-0-0] [lt=77] Slow write(duration=1189401, write_len=4096) [2024-04-24 10:00:50.713722] INFO [LIB] stat (utility.h:1197) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=94] [PALF STAT WRITE LOG COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=824497, this=0x14b0c0e353b0, extra_info="palf_id:1001") [2024-04-24 10:00:50.713809] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=101][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179748114}], scn_array=[{val:1713924048988077000, v:0}], curr_size=538, accum_size=2612, time_cost=1719871) [2024-04-24 10:00:50.713803] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=164][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:1002196895}], scn_array=[{val:1713924048914830000, v:0}], curr_size=122, accum_size=1695, time_cost=1609379) [2024-04-24 10:00:50.713843] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=114][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:79626650}], scn_array=[{val:1713924048914830000, v:0}], curr_size=122, accum_size=2362, time_cost=1637087) [2024-04-24 10:00:50.713854] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [874495][T1004_OB_SLOG][T1004][Y0-0000000000000000-0-0] [lt=112] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=93234, offset=21596662}, end_cursor:ObLogCursor{file_id=1, log_id=93235, offset=21596911}, is_inited:true, is_local:false, buf_size:8192, buf:0x14b19ef28050, len:1546, log_data_len:249, seq:93234, flush_finish:false, flush_ret:0}) [2024-04-24 10:00:50.713929] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=142][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:88123610}], scn_array=[{val:1713924048914830000, v:0}], curr_size=122, accum_size=1824, time_cost=1636704) [2024-04-24 10:00:50.713956] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=139] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=2612) [2024-04-24 10:00:50.713956] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=145] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=1695) [2024-04-24 10:00:50.713969] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=119] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=2362) [2024-04-24 10:00:50.713956] INFO [PALF] inner_write_once_ (log_block_handler.cpp:411) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=78] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=45718187, this={dio_aligned_buf:{buf_write_offset:3273, buf_padding_size:823, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:6151912, truncate_used_ts:7325559}, log_block_size:67108864, dir_fd:155, io_fd:157}, aligned_buf_len=4096, aligned_buf="Š€€€", aligned_block_offset=45715456, buf_len=542, total_write_size_=37002, total_write_size_after_dio_=290816, ob_pwrite_used_ts=5583625, count=63, each_pwrite_cost=88628) [2024-04-24 10:00:50.714008] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=94][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="10.0.0.3:2882", server="10.0.0.3:2882", update_func={old_end_lsn:{lsn:67159339}, new_end_lsn:{lsn:67159461}, old_advance_time_us:1713924048872238, new_ack_time_us:1713924050713998, advance delay(us):1841760}) [2024-04-24 10:00:50.714023] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=87] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=1824) [2024-04-24 10:00:50.714110] INFO [LIB] stat (utility.h:1214) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=98] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=124, this=0x14b1679f6c70, extra_info="") [2024-04-24 10:00:50.714123] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=102] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=488) [2024-04-24 10:00:50.714131] INFO [LIB] stat (utility.h:1214) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=122] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=226, this=0x14b176ffc530, extra_info="") [2024-04-24 10:00:50.714131] INFO [LIB] stat (utility.h:1197) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=142] [PALF STAT WRITE LOG COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1801964, this=0x14b14c9bd3b0, extra_info="palf_id:1") [2024-04-24 10:00:50.714123] INFO [LIB] stat (utility.h:1214) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=98] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=23735, this=0x14b1945fe530, extra_info="") [2024-04-24 10:00:50.714168] INFO [LIB] stat (utility.h:1214) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=120] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=212, this=0x14b176ffcc70, extra_info="") [2024-04-24 10:00:50.714205] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=61][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:314133163}], scn_array=[{val:1713924048910937000, v:0}], curr_size=542, accum_size=542, time_cost=1801964) [2024-04-24 10:00:50.714287] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=76] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=542) [2024-04-24 10:00:50.714305] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=164][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="10.0.0.3:2882", server="10.0.0.3:2882", update_func={old_end_lsn:{lsn:79626650}, new_end_lsn:{lsn:79626772}, old_advance_time_us:1713924048894851, new_ack_time_us:1713924050714294, advance delay(us):1819443}) [2024-04-24 10:00:50.714324] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=69] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=102, this=0x14b1470ab470, extra_info="") [2024-04-24 10:00:50.714351] INFO [LIB] stat (utility.h:1197) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=117] [PALF STAT WRITE LOG COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=909224, this=0x14b0f0aab3b0, extra_info="palf_id:1") [2024-04-24 10:00:50.714384] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=72] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=5753) [2024-04-24 10:00:50.714412] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=66] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=5, avg_log_batch_cnt=1, total_group_log_size=616, avg_group_log_size=123) [2024-04-24 10:00:50.714412] INFO [LIB] stat (utility.h:1214) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=81] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=368, this=0x14b157ff6530, extra_info="") [2024-04-24 10:00:50.714418] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=61][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:298522329}], scn_array=[{val:1713924048912363000, v:0}], curr_size=973, accum_size=2349, time_cost=1800961) [2024-04-24 10:00:50.714472] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=53] [PALF STAT FLUSH CB COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=153, this=0x14b0efb335e0, extra_info="palf_id:1") [2024-04-24 10:00:50.714485] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=61] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=2349) [2024-04-24 10:00:50.714506] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=60] [PALF STAT FLUSH CB COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=350, this=0x14b1470b55e0, extra_info="palf_id:1") [2024-04-24 10:00:50.714591] INFO [LIB] stat (utility.h:1214) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=60] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=116, this=0x14b1421a2530, extra_info="") [2024-04-24 10:00:50.714603] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=65][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1001, self="10.0.0.3:2882", server="10.0.0.3:2882", update_func={old_end_lsn:{lsn:88123610}, new_end_lsn:{lsn:88123732}, old_advance_time_us:1713924048894504, new_ack_time_us:1713924050714597, advance delay(us):1820093}) [2024-04-24 10:00:50.714635] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=78][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1001, self="10.0.0.3:2882", server="10.0.0.3:2882", update_func={old_end_lsn:{lsn:1002196895}, new_end_lsn:{lsn:1002197017}, old_advance_time_us:1713924048882614, new_ack_time_us:1713924050714629, advance delay(us):1832015}) [2024-04-24 10:00:50.714700] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=89] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="10.0.0.3:2882", committed size=1074) [2024-04-24 10:00:50.714738] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=95] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="10.0.0.3:2882", committed size=1695) [2024-04-24 10:00:50.714835] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=64] [PALF STAT FLUSH CB COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=214, this=0x14b0c0e355e0, extra_info="palf_id:1001") [2024-04-24 10:00:50.714864] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=97] [PALF STAT FLUSH CB COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=155, this=0x14b1336f55e0, extra_info="palf_id:1001") [2024-04-24 10:00:50.715017] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [874529][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=60] cannot_recycle_log_size statistics(cannot_recycle_log_size=208104929, threshold=10436770529, need_update_checkpoint_scn=false) [2024-04-24 10:00:50.715066] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [874740][T1003_ObTimer][T1003][Y0-0000000000000000-0-0] [lt=68] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-04-24 10:00:50.715165] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [874740][T1003_ObTimer][T1003][Y0-0000000000000000-0-0] [lt=66] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-04-24 10:00:50.715238] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [874740][T1003_ObTimer][T1003][Y0-0000000000000000-0-0] [lt=66] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=9338) [2024-04-24 10:00:50.715489] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=99][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="10.0.0.3:2882", server="10.0.0.3:2882", update_func={old_end_lsn:{lsn:314133163}, new_end_lsn:{lsn:314133705}, old_advance_time_us:1713924048878310, new_ack_time_us:1713924050715482, advance delay(us):1837172}) [2024-04-24 10:00:50.715562] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=62] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=664) [2024-04-24 10:00:50.715761] INFO [LIB] stat (utility.h:1197) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=69] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=91, this=0x14b14c9b3470, extra_info="") [2024-04-24 10:00:50.715931] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=141] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=2, avg_log_batch_cnt=1, total_group_log_size=1150, avg_group_log_size=575) [2024-04-24 10:00:50.716031] INFO [LIB] stat (utility.h:1197) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=67] [PALF STAT FLUSH CB COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=555, this=0x14b14c9bd5e0, extra_info="palf_id:1") [2024-04-24 10:00:50.716367] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=222][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="10.0.0.3:2882", server="10.0.0.3:2882", update_func={old_end_lsn:{lsn:298522329}, new_end_lsn:{lsn:298523302}, old_advance_time_us:1713924048892947, new_ack_time_us:1713924050716357, advance delay(us):1823410}) [2024-04-24 10:00:50.716451] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=78] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=1918) [2024-04-24 10:00:50.716805] INFO [LIB] stat (utility.h:1197) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=115] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=97, this=0x14b0f0aa1470, extra_info="") [2024-04-24 10:00:50.716953] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=120] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=2, avg_log_batch_cnt=2, total_group_log_size=2067, avg_group_log_size=1033) [2024-04-24 10:00:50.717061] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [862230][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=78] Slow write(duration=1190356, write_len=4096) [2024-04-24 10:00:50.717079] INFO [LIB] stat (utility.h:1197) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=74] [PALF STAT FLUSH CB COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=374, this=0x14b0f0aab5e0, extra_info="palf_id:1") [2024-04-24 10:00:50.717127] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [862230][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=60] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=101081, offset=22880392}, end_cursor:ObLogCursor{file_id=1, log_id=101082, offset=22880641}, is_inited:true, is_local:false, buf_size:8192, buf:0x14b19ef3c050, len:3960, log_data_len:249, seq:101081, flush_finish:false, flush_ret:0}) [2024-04-24 10:00:50.717510] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [862263][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=112] cannot_recycle_log_size statistics(cannot_recycle_log_size=898934974, threshold=1739461755, need_update_checkpoint_scn=false) [2024-04-24 10:00:50.718232] INFO [LIB] stat (utility.h:1197) [861568][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=80] [PALF STAT FLUSH CB COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=148, this=0x14b186b515e0, extra_info="palf_id:1") [2024-04-24 10:00:50.721915] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=70] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8890687488, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:50.732241] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=72][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.733556] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=101] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.738950] INFO do_work (ob_rl_mgr.cpp:709) [861413][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=163] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-24 10:00:50.749387] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=155] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.755254] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:426) [862397][T1001_FreInfoRe][T1001][Y0-0000000000000000-0-0] [lt=1103] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2024-04-24 10:00:50.757471] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=101] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.760247] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [874764][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2634-0-0] [lt=62][errcode=-4023] tx is 2pc logging(this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:50.760419] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [874764][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2634-0-0] [lt=164][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fc095050, ref:3, trans_id:{txid:338713}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924078889979, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048912965], ctx_create_time:1713924048898263}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909713, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338713}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048912363000, v:0}, start_scn:{val:1713924048912363000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924048912363000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338713} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3595883536 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048912363000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048912363, block_frozen_memtable:null}) [2024-04-24 10:00:50.760518] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [874764][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2634-0-0] [lt=92][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338713}) [2024-04-24 10:00:50.760585] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [874764][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2634-0-0] [lt=59][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924050755626, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:50.760715] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [874764][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2634-0-0] [lt=93][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338713}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924050755626, cluster_id:1713874979}, expire_ts:1713924078907999, commit_start_scn:{val:1713924048906163000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:50.762000] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [861591][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=49] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-04-24 10:00:50.762113] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [861591][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=106] start lib cache evict(tenant_id=1, mem_hold=85983232, mem_limit=107374180, cache_obj_num=158, cache_node_num=157) [2024-04-24 10:00:50.762183] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [861591][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=61] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=85983232, mem_limit=107374180, cache_obj_num=158, cache_node_num=157) [2024-04-24 10:00:50.762291] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [861591][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=99] schedule next cache evict task(evict_interval=5000000) [2024-04-24 10:00:50.763214] INFO [STORAGE.TRANS] retransmit_downstream_msg_ (ob_two_phase_upstream_committer.cpp:311) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=144] unresponded participant(i=1, *this={this:0x14b1481c0010, ref:8, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048664959], ctx_create_time:1713924045155303}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79589464}, {lsn:79618713}], redo_log_no:2, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:6, data_.length():1313}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048664122000, v:0}, trans_type:2, next_log_entry_no:4, max_applied_log_ts:{val:1713924048664122000, v:0}, max_applying_log_ts:{val:1713924048664122000, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:1713924048659397, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:79620195}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048272353001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877443458344 max_table_version=1713877467338312 trans_version={val:1713924048664122000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1 callback_alloc_count=134 callback_free_count=0 checksum=141527899 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=158 redo_sync_succ_count=158 redo_sync_fail_count=0 main_list_length=158 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=158, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924048272353001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048665345, block_frozen_memtable:null}) [2024-04-24 10:00:50.763476] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=183] handle timeout(ret=0, *this={this:0x14b1481c0010, ref:9, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048664959], ctx_create_time:1713924045155303}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79589464}, {lsn:79618713}], redo_log_no:2, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:6, data_.length():1313}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048664122000, v:0}, trans_type:2, next_log_entry_no:4, max_applied_log_ts:{val:1713924048664122000, v:0}, max_applying_log_ts:{val:1713924048664122000, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:1713924048659397, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:79620195}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048272353001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877443458344 max_table_version=1713877467338312 trans_version={val:1713924048664122000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1 callback_alloc_count=134 callback_free_count=0 checksum=141527899 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=158 redo_sync_succ_count=158 redo_sync_fail_count=0 main_list_length=158 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=158, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924048272353001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048665345, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=800000) [2024-04-24 10:00:50.763696] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=199] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:50.763918] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=91] handle timeout(ret=0, *this={this:0x14b055e1e050, ref:4, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924048589222}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88120890}], redo_log_no:1, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048658150001, v:0}, trans_type:2, next_log_entry_no:3, max_applied_log_ts:{val:1713924048658150001, v:0}, max_applying_log_ts:{val:1713924048658150001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:88121631}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048588043001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=0 trans_version={val:1713924048658150001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=1 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924048588043001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048589222, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=800000) [2024-04-24 10:00:50.764075] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=139] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:50.765637] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [861591][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=114] Dumping All Cache Objs(alloc_obj_list.count()=158, alloc_obj_list=[{obj_id:108, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176332}, {obj_id:987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:990, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:1131, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182834}, {obj_id:1152, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178656}, {obj_id:1322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193763}, {obj_id:1761, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:1763, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:2015, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:2042, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7725, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190714}, {obj_id:7765, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189565}, {obj_id:7797, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196189}, {obj_id:7805, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188533}, {obj_id:7824, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188030}, {obj_id:7830, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198488}, {obj_id:7833, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179266}, {obj_id:8079, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174033}, {obj_id:8170, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175223}, {obj_id:8265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:238685}, {obj_id:9675, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9676, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174121}, {obj_id:9679, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194296}, {obj_id:9685, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189832}, {obj_id:9686, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:9689, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174039}, {obj_id:9691, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177545}, {obj_id:9692, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174076}, {obj_id:9693, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174036}, {obj_id:9698, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194888}, {obj_id:9700, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197705}, {obj_id:9701, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9706, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174016}, {obj_id:9708, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204160}, {obj_id:9709, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188762}, {obj_id:9710, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184970}, {obj_id:9711, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183911}, {obj_id:9712, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9713, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174856}, {obj_id:9714, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217681}, {obj_id:9715, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172224}, {obj_id:9716, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180621}, {obj_id:9722, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191631}, {obj_id:9727, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180931}, {obj_id:9733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:205820}, {obj_id:9737, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175596}, {obj_id:9738, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176715}, {obj_id:9739, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179722}, {obj_id:9740, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178553}, {obj_id:9741, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157777}, {obj_id:9742, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179253}, {obj_id:9743, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183140}, {obj_id:9744, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181024}, {obj_id:9745, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191526}, {obj_id:9746, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213392}, {obj_id:9747, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203551}, {obj_id:9748, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184694}, {obj_id:9753, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185653}, {obj_id:9755, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179890}, {obj_id:9757, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173875}, {obj_id:9759, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172928}, {obj_id:9767, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174617}, {obj_id:9769, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195084}, {obj_id:9770, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174535}, {obj_id:9771, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174529}, {obj_id:9772, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174532}, {obj_id:9773, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175422}, {obj_id:9805, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179080}, {obj_id:9806, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180315}, {obj_id:9807, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9829, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157844}, {obj_id:9830, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9831, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182834}, {obj_id:9832, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263635}, {obj_id:9833, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202924}, {obj_id:9834, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182376}, {obj_id:9835, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:159523}, {obj_id:9838, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:9840, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176677}, {obj_id:9841, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185218}, {obj_id:9842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184511}, {obj_id:9863, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174701}, {obj_id:9869, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9870, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:267554}, {obj_id:9885, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:9894, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179593}, {obj_id:9895, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181903}, {obj_id:9896, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191549}, {obj_id:9897, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265843}, {obj_id:9930, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185744}, {obj_id:10010, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174572}, {obj_id:10028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175408}, {obj_id:10029, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175408}, {obj_id:10030, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:10162, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:10163, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191816}, {obj_id:10165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184764}, {obj_id:10192, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207381}, {obj_id:10210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177328}, {obj_id:10212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183041}, {obj_id:10213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210106}, {obj_id:10215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204324}, {obj_id:10216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179228}, {obj_id:10233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:218946}, {obj_id:10234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194032}, {obj_id:10253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202373}, {obj_id:10302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:10303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181684}, {obj_id:10305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176376}, {obj_id:10432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194680}, {obj_id:10456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188728}, {obj_id:10457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:231788}, {obj_id:10458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:227193}, {obj_id:10459, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:10542, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198627}, {obj_id:26329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174701}, {obj_id:34021, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172632}, {obj_id:34022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174307}, {obj_id:34023, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172632}, {obj_id:34034, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172632}, {obj_id:34035, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172632}, {obj_id:34841, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:34842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:34843, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:166006}, {obj_id:34844, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:162329}, {obj_id:34845, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:36803, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210727}, {obj_id:36804, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215815}, {obj_id:37658, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208246}, {obj_id:37660, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209383}, {obj_id:37662, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199145}, {obj_id:37725, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:276579}, {obj_id:37727, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206681}, {obj_id:37728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203936}, {obj_id:37729, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174864}, {obj_id:37732, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177206}, {obj_id:37738, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:37739, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179663}, {obj_id:37741, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:37889, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234233}, {obj_id:37890, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:256044}, {obj_id:37891, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232152}, {obj_id:37892, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300067}, {obj_id:37893, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:305590}, {obj_id:37904, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271171}, {obj_id:38027, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210421}, {obj_id:38106, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:38107, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:166011}, {obj_id:38108, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161304}, {obj_id:38109, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175592}, {obj_id:38110, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175634}, {obj_id:38124, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175648}, {obj_id:38482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157800}, {obj_id:38957, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208649}, {obj_id:39281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209017}, {obj_id:39282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210259}, {obj_id:39314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:205451}]) [2024-04-24 10:00:50.765943] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [861591][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=294] schedule next cache evict task(evict_interval=5000000) [2024-04-24 10:00:50.766658] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=74][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.766702] INFO [STORAGE.TRANS] handle_2pc_commit_request_impl_ (ob_two_phase_downstream_committer.cpp:420) [2319281][T1004_L0_G0][T1004][Y0-0000000000000000-0-0] [lt=95] committer is under logging(ret=0, *this={this:0x14b055e1e050, ref:4, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924048589222}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88120890}], redo_log_no:1, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048658150001, v:0}, trans_type:2, next_log_entry_no:3, max_applied_log_ts:{val:1713924048658150001, v:0}, max_applying_log_ts:{val:1713924048658150001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:88121631}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048588043001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=0 trans_version={val:1713924048658150001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=1 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924048588043001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048589222, block_frozen_memtable:null}) [2024-04-24 10:00:50.767472] INFO [STORAGE.TRANS] handle_2pc_commit_request_impl_ (ob_two_phase_downstream_committer.cpp:420) [2319281][T1004_L0_G0][T1004][Y0-0000000000000000-0-0] [lt=205] committer is under logging(ret=0, *this={this:0x14b055e1e050, ref:4, trans_id:{txid:176888}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924048589222}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:40, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88120890}], redo_log_no:1, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048658150001, v:0}, trans_type:2, next_log_entry_no:3, max_applied_log_ts:{val:1713924048658150001, v:0}, max_applying_log_ts:{val:1713924048658150001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:88121631}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048588043001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=0 trans_version={val:1713924048658150001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=1 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:50, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924048588043001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048589222, block_frozen_memtable:null}) [2024-04-24 10:00:50.779515] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=60] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.779596] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=74] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.781713] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=115] new mysql sessid created(conn.sessid_=3221542446, support_ssl=false) [2024-04-24 10:00:50.781918] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=106] sm conn init succ(conn.sessid_=3221542446, sess.client_addr_="10.0.0.3:48140") [2024-04-24 10:00:50.782036] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=83] accept one succ(*s={this:0x14b11d03e030, session_id:3221542446, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:50.782363] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:943) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=90][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x14b11d03e030, session_id:3221542446, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:50.782479] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=96] kill and revert session(conn.sessid_=3221542446, proxy_sessid=0, server_id=1, ret=0) [2024-04-24 10:00:50.782597] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:918) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=110] sock ref clean, do destroy(*s={this:0x14b11d03e030, session_id:3221542446, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:50.782714] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:223) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=107] mark session id unused(conn.sessid_=3221542446) [2024-04-24 10:00:50.782783] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=59] connection close(sessid=3221542446, proxy_sessid=0, tenant_id=0, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:0], disconnect_state=0) [2024-04-24 10:00:50.785727] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [862534][T1002_L0_G5][T1002][YB420A000002-000616C2BEDDD129-0-0] [lt=142] is logging [2024-04-24 10:00:50.785834] WDIAG [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:116) [862534][T1002_L0_G5][T1002][YB420A000002-000616C2BEDDD129-0-0] [lt=83][errcode=-4023] submit log callback use too mush time(submit_log_ts=1713924049918819, cb={log_ts:{val:1713924049917601001, v:0}, log_buf:"", pos:40, id_srv_type:0, limited_id:1713924066491232000}, service_type=0, latest_log_ts={val:1713924039886464001, v:0}, self="10.0.0.3:2882") [2024-04-24 10:00:50.785957] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [862534][T1002_L0_G5][T1002][YB420A000002-000616C2BEDDD129-0-0] [lt=117] submit log with lock(ret=-4023, service_type_=0, last_id=1713924066491232000, limited_id=1713924066491232000) [2024-04-24 10:00:50.789796] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:426) [861646][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=80] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2024-04-24 10:00:50.797240] INFO [SQL.JO] compute_table_location (ob_join_order.cpp:278) [861895][T1_L0_G0][T1][YB420A000003-000616C2ACE8FC9E-0-0] [lt=92] succeed to calculate base table sharding info(table_id=18446744073709551614, ref_table_id=104, is_global_index=false) [2024-04-24 10:00:50.797344] INFO [SQL.JO] init_est_sel_info_for_access_path (ob_join_order.cpp:12189) [861895][T1_L0_G0][T1][YB420A000003-000616C2ACE8FC9E-0-0] [lt=85] total rowcount, use statistics(table_meta_info_.table_row_count_=10, table_meta_info_.average_row_size_=2.060000000000000000e+02, table_meta_info_.micro_block_count_=1, table_meta_info_.part_size_=2.060000000000000000e+03) [2024-04-24 10:00:50.807246] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=97] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.807375] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=100] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.807460] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=79] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.810020] INFO [MDS.EVENT]try_recycle (mds_table_impl.ipp:1418) [874394][T1004_Occam][T1004][YB420A000003-000616C2B6C982EA-0-0] [lt=140] RECYCLE(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:1152921504606847581}}, event={alloc:null, timestamp:"2024-04-24 10:00:50.810007", event:"RECYCLE", info_str:"recycle_scn:{val:1, v:0}", unit_id:255, key_str:"", writer_type:0, writer_id:0, seq_no:0, redo_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, trans_version:{val:18446744073709551615, v:3}, node_type:0, state:5}) [2024-04-24 10:00:50.811042] INFO [MDS.EVENT]try_recycle (mds_table_impl.ipp:1418) [874394][T1004_Occam][T1004][YB420A000003-000616C2B6C982EA-0-0] [lt=159] RECYCLE(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:200317}}, event={alloc:null, timestamp:"2024-04-24 10:00:50.811036", event:"RECYCLE", info_str:"recycle_scn:{val:1, v:0}", unit_id:255, key_str:"", writer_type:0, writer_id:0, seq_no:0, redo_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, trans_version:{val:18446744073709551615, v:3}, node_type:0, state:5}) [2024-04-24 10:00:50.816009] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [874394][T1004_Occam][T1004][YB420A000003-000616C2B6C982EA-0-0] [lt=162] for each ls(succ_num=3, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:50.832433] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=96][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.832754] WDIAG [STORAGE.TRANS] inner_lock_for_read (ob_tx_data_functor.cpp:290) [2323137][T1004_L0_G0][T1004][YB420A000002-000616C2B2CC0A33-0-0] [lt=71][errcode=-6004] lock_for_read need retry(ret=-6004, tx_data={tx_id:{txid:176888}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048272353001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}, lock_for_read_arg_={mvcc_acc_ctx_:{type:1, abs_lock_timeout:1713924148785466, tx_lock_timeout:-1, snapshot:{version:{val:1713924048839739000, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x14b0efafe490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, data_trans_id_:{txid:176888}, data_sql_sequence_:1713924048475558, read_latest_:false, scn_:{val:1713924048664122000, v:0}}, tx_cc_ctx={state:40, prepare_version:{val:1713924048664122000, v:0}}) [2024-04-24 10:00:50.833803] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=149] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.837740] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [861895][T1_L0_G0][T1][YB420A000003-000616C2ACE8FC9E-0-0] [lt=103][errcode=0] set logical del time(cache_obj->get_logical_del_time()=576363411853, cache_obj->added_lc()=false, cache_obj->get_object_id()=39599, cache_obj->get_tenant_id()=1, lbt()="0x121f36ac 0xc327dd4 0x4e3fdbe 0x4e31e28 0x4e29b67 0x4e21201 0x4e1eb17 0x4e0e175 0xa7ac354 0x12d1fdbc 0x14b1c1d132de 0x14b1c1a44e83") [2024-04-24 10:00:50.838722] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [861692][T1_LSArchiveMgr][T1][YB420A000003-000616C2A888F0E7-0-0] [lt=162] gc stale ls task succ [2024-04-24 10:00:50.840493] INFO [CLOG] is_in_sync (ob_log_handler.cpp:546) [861741][T1_L0_G0][T1][YB420A000002-000616C2A8DC429D-0-0] [lt=63] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2024-04-24 10:00:50.841098] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_ls_info.cpp:54) [861741][T1_L0_G0][T1][YB420A000002-000616C2A8DC429D-0-0] [lt=92][errcode=-4008] execute fail(ret=-4008) [2024-04-24 10:00:50.841786] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=100] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-04-24 10:00:50.849694] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=104] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.857720] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=76] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.864173] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B8-0-0] [lt=211][errcode=-4023] tx is 2pc logging(this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:50.864336] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B8-0-0] [lt=157][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e38050, ref:3, trans_id:{txid:176913}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048509205, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048839935], ctx_create_time:1713924048830281}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048835083, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176913}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048839739000, v:0}, start_scn:{val:1713924048839739000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1713877454130248 trans_version={val:1713924048839739000, v:0} commit_version={val:1713924048839739000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176913} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=1462026768 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048839739000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048839739, block_frozen_memtable:null}) [2024-04-24 10:00:50.864484] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B8-0-0] [lt=138][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176913}) [2024-04-24 10:00:50.864605] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B8-0-0] [lt=62][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924050859005, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:50.864750] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B8-0-0] [lt=126][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176913}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924050859005, cluster_id:1713874979}, expire_ts:1713925048510011, commit_start_scn:{val:1713924048832745000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:50.866834] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=66][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.879738] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=125] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.879927] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=174] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.888163] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1457) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2B9DC616B-0-0] [lt=79] receive register_req from acceptor, ignore(self="10.0.0.3:2882", palf_id=1001, child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924050883544, keepalive_ts:-1}) [2024-04-24 10:00:50.888327] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1476) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2B9DC616B-0-0] [lt=154] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="10.0.0.3:2882", child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924050883544, keepalive_ts:-1}, is_to_leader=true) [2024-04-24 10:00:50.897075] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [861461][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=107] skip inspect bad block(last_check_time=1713875167993818, last_macro_idx=-1) [2024-04-24 10:00:50.897301] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2B79C2373-0-0] [lt=80] current clock is (clock_=0, max_seq_=0, mem_limit=3181127926, cur_mem_hold=297795584, attr_.tenant_id_=1004) [2024-04-24 10:00:50.897614] INFO [LIB] stat (utility.h:1197) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2B79C2373-0-0] [lt=69] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=70, this=0x14b0efb29470, extra_info="") [2024-04-24 10:00:50.897773] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2B79C2373-0-0] [lt=125] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=3, avg_log_batch_cnt=1, total_group_log_size=2860, avg_group_log_size=953) [2024-04-24 10:00:50.897951] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2B79C2373-0-0] [lt=147] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=2484) [2024-04-24 10:00:50.898055] INFO [LIB] stat (utility.h:1197) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2B79C2373-0-0] [lt=85] [PALF STAT APPEND COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=0, this=0x14b0efafce50, extra_info="id:1") [2024-04-24 10:00:50.907378] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=101] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.907464] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=71] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.907594] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=116] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.911555] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [862405][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=108] handle timeout(ret=0, *this={this:0x14b15a16b090, ref:4, trans_id:{txid:568574}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924078927990, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048929680], ctx_create_time:1713924048928190}{ls_id:{id:1}, session_id:3221542296, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.3:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048917318, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048929552000, v:0}, log_ts:{val:1713924048929552000, v:0}, lsn:{lsn:314133761}, submit_ts:1713924048929552}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:568574}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:568574}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048929552000, v:0}, start_scn:{val:1713924048929552000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875429046312 max_table_version=1713875429046312 trans_version={val:1713924048929552000, v:0} commit_version={val:1713924048929552000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:568574} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=593250361 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048929552000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048929552, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=400000) [2024-04-24 10:00:50.911765] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [862405][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=200] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1713924048929552000, v:0}, log_ts:{val:1713924048929552000, v:0}, lsn:{lsn:314133761}, submit_ts:1713924048929552}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:568574}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924048929552000, v:0}, log_ts:{val:1713924048929552000, v:0}, lsn:{lsn:314133761}, submit_ts:1713924048929552}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:568574}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:50.918543] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [861742][BlackListServic][T1][YB420A000003-000616C2A8E93205-0-0] [lt=83] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12239, add_timestamp:1713924050918455}) [2024-04-24 10:00:50.918781] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [861459][VTblLocAsyncUp0][T0][YB420A000003-000616C2AB69B993-0-0] [lt=83] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12239, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"10.0.0.3:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1713924050918688, sql_renew_time:1713924050918688, is_mark_fail:false}]) [2024-04-24 10:00:50.918933] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [861459][VTblLocAsyncUp0][T0][YB420A000003-000616C2AB69B993-0-0] [lt=143] success to process renew task(task={tenant_id:1, table_id:12239, add_timestamp:1713924050918455}, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"10.0.0.3:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1713924050918688, sql_renew_time:1713924050918688, is_mark_fail:false}]) [2024-04-24 10:00:50.921688] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=95] gts statistics(tenant_id=1, gts_rpc_cnt=17, get_gts_cache_cnt=152, get_gts_with_stc_cnt=7, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-04-24 10:00:50.921803] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=88] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1713924050420020], gts:1713924050416597000, latest_srr:[mts=1713924050921569]}) [2024-04-24 10:00:50.922202] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=330] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-24 10:00:50.922319] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [861494][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=79] ts response task statistics(alloc_count=2169846, free_count=1084926) [2024-04-24 10:00:50.922472] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [861494][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=81] is logging [2024-04-24 10:00:50.922556] WDIAG [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:116) [861494][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=62][errcode=-4023] submit log callback use too mush time(submit_log_ts=1713924049918819, cb={log_ts:{val:1713924049917601001, v:0}, log_buf:"", pos:40, id_srv_type:0, limited_id:1713924066491232000}, service_type=0, latest_log_ts={val:1713924039886464001, v:0}, self="10.0.0.3:2882") [2024-04-24 10:00:50.922669] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [861494][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=101] submit log with lock(ret=-4023, service_type_=0, last_id=1713924066491232000, limited_id=1713924066491232000) [2024-04-24 10:00:50.922689] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=78] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8882298880, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:50.924769] INFO [PALF] get_log (palf_handle_impl.cpp:4002) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2B1BC330F-0-0] [lt=106] submit fetch log task success(ret=0, palf_id=1, server="10.0.0.2:2882", fetch_type=0, prev_lsn={lsn:314133041}, start_lsn={lsn:314133163}, fetch_log_size=41943040, fetch_log_count=2048, accepted_mode_pid=5) [2024-04-24 10:00:50.924927] INFO [PALF] handle (fetch_log_engine.cpp:284) [862323][T1001_FetchLog0][T1001][Y0-0000000000000000-0-0] [lt=70] handle fetch_log_task(fetch_log_task={timestamp_us:1713924050924741, id:1, server:"10.0.0.2:2882", fetch_type:"FETCH_LOG_FOLLOWER", proposal_id:5, prev_lsn:{lsn:314133041}, start_lsn:{lsn:314133163}, log_size:41943040, log_count:2048, accepted_mode_pid:5}) [2024-04-24 10:00:50.924939] INFO pn_ratelimit (group.c:551) [861521][IngressService][T0][Y0-0000000000000000-0-0] [lt=71] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-04-24 10:00:50.925100] INFO [PALF] read (log_cache.cpp:100) [862323][T1001_FetchLog0][T1001][Y0-0000000000000000-0-0] [lt=115] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3804, hit_cnt=2, read_cnt=2, hit rate=1.000000000000000000e+00) [2024-04-24 10:00:50.925295] WDIAG [PALF] batch_fetch_log_each_round_ (palf_handle_impl.cpp:3776) [862323][T1001_FetchLog0][T1001][Y0-0000000000000000-0-0] [lt=87][errcode=-4008] iterator next failed(ret=-4008, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, iterator={iterator_impl:{buf_:0x14b13e1b9000, next_round_pread_size:3678208, curr_read_pos:542, curr_read_buf_start_pos:0, curr_read_buf_end_pos:542, log_storage_:{IteratorStorage:{start_lsn:{lsn:314133163}, end_lsn:{lsn:314133705}, read_buf:{buf_len_:8192, buf_:0x14b13e1b9000}, block_size:67104768, log_storage_:0x14b14c9b9470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713924048910937000, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:1, group_size:486, proposal_id:5, committed_lsn:{lsn:314133163}, max_scn:{val:1713924048910937000, v:0}, accumulated_checksum:2907004606, log_id:505464, flag:1}}, init_mode_version:0, accumulate_checksum:2907004606, curr_entry_is_padding:0, padding_entry_size:486, padding_entry_scn:{val:1713924048910937000, v:0}}}, replayable_point={val:18446744073709551615, v:3}, prev_lsn={lsn:314133163}) [2024-04-24 10:00:50.925475] INFO [PALF] fetch_log_from_storage_ (palf_handle_impl.cpp:3713) [862323][T1001_FetchLog0][T1001][Y0-0000000000000000-0-0] [lt=86] fetch_log_from_storage_ finished(ret=0, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, prev_lsn={lsn:314133041}, iterator={iterator_impl:{buf_:0x14b13e1b9000, next_round_pread_size:3678208, curr_read_pos:542, curr_read_buf_start_pos:0, curr_read_buf_end_pos:542, log_storage_:{IteratorStorage:{start_lsn:{lsn:314133163}, end_lsn:{lsn:314133705}, read_buf:{buf_len_:8192, buf_:0x14b13e1b9000}, block_size:67104768, log_storage_:0x14b14c9b9470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713924048910937000, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:1, group_size:486, proposal_id:5, committed_lsn:{lsn:314133163}, max_scn:{val:1713924048910937000, v:0}, accumulated_checksum:2907004606, log_id:505464, flag:1}}, init_mode_version:0, accumulate_checksum:2907004606, curr_entry_is_padding:0, padding_entry_size:486, padding_entry_scn:{val:1713924048910937000, v:0}}}, fetch_log_count=2048, has_fetched_log_count=1) [2024-04-24 10:00:50.925643] INFO [PALF] submit_fetch_mode_meta_resp (log_mode_mgr.cpp:826) [862323][T1001_FetchLog0][T1001][Y0-0000000000000000-0-0] [lt=108] submit_change_mode_meta_req success(ret=0, palf_id=1, server="10.0.0.2:2882", mode_meta={version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_pid=5) [2024-04-24 10:00:50.925737] INFO [PALF] fetch_log_from_storage (palf_handle_impl.cpp:3498) [862323][T1001_FetchLog0][T1001][Y0-0000000000000000-0-0] [lt=87] [PALF STAT FETCH LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=41943040) [2024-04-24 10:00:50.925896] INFO [LIB] stat (utility.h:1197) [862323][T1001_FetchLog0][T1001][Y0-0000000000000000-0-0] [lt=91] [PALF STAT FETCH LOG TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=112, this=0x14b157ff2600, extra_info="") [2024-04-24 10:00:50.925993] INFO [LIB] stat (utility.h:1197) [862323][T1001_FetchLog0][T1001][Y0-0000000000000000-0-0] [lt=85] [PALF STAT FETCH LOG EXECUTE COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=986, this=0x14b157ff2830, extra_info="") [2024-04-24 10:00:50.926107] INFO [PALF] handle (fetch_log_engine.cpp:321) [862323][T1001_FetchLog0][T1001][Y0-0000000000000000-0-0] [lt=102] handle fetch log task(ret=0, palf_id=1, handle_cost_time_us=986, fetch_log_task={timestamp_us:1713924050924741, id:1, server:"10.0.0.2:2882", fetch_type:"FETCH_LOG_FOLLOWER", proposal_id:5, prev_lsn:{lsn:314133041}, start_lsn:{lsn:314133163}, log_size:41943040, log_count:2048, accepted_mode_pid:5}, fetch_stat={total_size:542, group_log_cnt:1, read_cost:0, get_cost:0, send_cost:80}) [2024-04-24 10:00:50.928544] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C48-0-0] [lt=81][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="10.0.0.3:2882", server="10.0.0.2:2882", update_func={old_end_lsn:{lsn:67159339}, new_end_lsn:{lsn:67159461}, old_advance_time_us:1713924048826492, new_ack_time_us:1713924050928530, advance delay(us):2102038}) [2024-04-24 10:00:50.928777] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C48-0-0] [lt=80] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=100, this=0x14b1470ab6a0, extra_info="palf_id:1") [2024-04-24 10:00:50.928938] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C48-0-0] [lt=140] [PALF STAT LOG LIFE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=2025028, this=0x14b1470ab8d0, extra_info="palf_id:1") [2024-04-24 10:00:50.928991] WDIAG [CLOG] statistics_cb_cost_ (ob_log_apply_service.cpp:933) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=108][errcode=-4389] cb cost too much time(lsn={lsn:67159395}, scn={val:1713924048836268000, v:0}, idx=0, total_cost_time=2024771, append_cost_time=253, cb_wait_thread_time=2025081, cb_wait_commit_time=0, cb_cost_time=-563, append_start_time=1713924048903596, append_finish_time=1713924048903849, cb_first_handle_time=1713924050928930, cb_finish_time=1713924050928367) [2024-04-24 10:00:50.929044] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C48-0-0] [lt=100] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=1, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=209, avg_log_submit_to_flush_time=1810167, avg_log_submit_to_slide_time=2024819) [2024-04-24 10:00:50.929126] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C48-0-0] [lt=72][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=259318, log_task={header:{begin_lsn:{lsn:67159339}, end_lsn:{lsn:67159461}, log_id:259318, min_scn:{val:1713924048836268000, v:0}, max_scn:{val:1713924048836268000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:67159339}, data_checksum:667309872, accum_checksum:907871364, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924048903610, freeze_ts:1713924048903618, submit_ts:1713924048903819, flushed_ts:1713924050713986, gen_to_freeze cost time:8, gen_to_submit cost time:209, submit_to_flush cost time:1810167}, fs_cb_begin_ts=1713924050928638, log_life_time=2025028) [2024-04-24 10:00:50.929550] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=133] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60476}, multi_version_start=1713895314983386001, old_min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, old_tablet_multi_version_start=1713895214593329000, old_tablet_snaphsot_version=1713895314983386001, tablet={this:0x14b0dfe21b60, is_inited:true, wash_score:1713924048462342000, hold_ref_cnt:true, ref_cnt:2, version:2, length:0, tablet_addr:[2602-2048-0-2305843611584950272-0](offset=2093056,size=561,type=2,seq=0), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:60476}, data_tablet_id:{id:476}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1713895314983386001, multi_version_start:1713895214593329000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1713895200575122000, cur_report_version:1713895200575122000, data_checksum:0, row_count:0}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1713877340363486001, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x14b0dfe221f4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:0}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[2602-2048-0-2305843315232198656-0](offset=2084864,size=285,type=2,seq=0), type:"MAJOR", tablet_id:{id:60476}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1713895200575122000, v:0}}, snapshot_version:1713895200575122000, max_merge_version:1713895200575122000}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{null}}}, addr:[2602-2048-0-2305843250807693312-0](offset=2088960,size=225,type=2,seq=0)}, storage_schema_addr:{ptr:null, addr:[2602-2048-0-2305843163834605793-0](offset=2089185,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[2602-2048-0-2305843251881435505-0](offset=2089329,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1713895200575122000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1713877339311783001, v:0}, create_commit_version:1713877339311783001, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0}) [2024-04-24 10:00:50.929638] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD295-0-0] [lt=60][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1001, self="10.0.0.3:2882", server="10.0.0.2:2882", update_func={old_end_lsn:{lsn:88121908}, new_end_lsn:{lsn:88123610}, old_advance_time_us:1713924048830771, new_ack_time_us:1713924050929627, advance delay(us):2098856}) [2024-04-24 10:00:50.929720] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=163] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60476}, multi_version_start=1713895314983386001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.929771] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD295-0-0] [lt=75] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=24, this=0x14b0c0e2b6a0, extra_info="palf_id:1001") [2024-04-24 10:00:50.929856] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD295-0-0] [lt=70] [PALF STAT LOG LIFE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=634648, this=0x14b0c0e2b8d0, extra_info="palf_id:1001") [2024-04-24 10:00:50.929926] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2A1-0-0] [lt=88][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048823139000, v:0}, log_ts:{val:1713924048823139000, v:0}, lsn:{lsn:88121964}, submit_ts:1713924048823139}, this:0x14b1481b8b40, is_inited_:true, trans_id:{txid:327197}, ls_id:{id:1001}, ctx:0x14b1481b5bd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=2106757, ctx_lock_wait_time=10) [2024-04-24 10:00:50.929980] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=69][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048823139001, v:0}, lsn:{lsn:88122845}, submit_ts:1713924048840938}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=2089029, ctx_lock_wait_time=7) [2024-04-24 10:00:50.930017] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD295-0-0] [lt=152] [PALF STAT LOG TASK TIME](palf_id=1001, self="10.0.0.3:2882", total_slide_log_cnt=4, avg_log_gen_to_freeze_time=16384, avg_log_gen_to_submit_time=16569, avg_log_submit_to_flush_time=34230, avg_log_submit_to_slide_time=618078) [2024-04-24 10:00:50.930099] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD295-0-0] [lt=60][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=330732, log_task={header:{begin_lsn:{lsn:88121908}, end_lsn:{lsn:88123610}, log_id:330732, min_scn:{val:1713924048823139000, v:0}, max_scn:{val:1713924048823139001, v:0}, data_len:1646, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:88121908}, data_checksum:955279408, accum_checksum:2323250645, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924048823990, freeze_ts:1713924048881514, submit_ts:1713924048881705, flushed_ts:1713924048894478, gen_to_freeze cost time:57524, gen_to_submit cost time:57715, submit_to_flush cost time:12773}, fs_cb_begin_ts=1713924050929720, log_life_time=2105730) [2024-04-24 10:00:50.930090] INFO [MDS] on_commit (buffer_ctx.h:73) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=77] buffer ctx on commit(this=0x14b055e204c0, *this={this:0x14b055e204c0, ctx:0x14b053e0b080, ctx:{writer:{writer_type:"TRANS", writer_id:176888}, write_list:list_head:0x14b0d6aad380, list_tail:0x14b0d6aad750, list:[this:0x14b0d6aad380, ls_id:{id:1001}, tablet_id:{id:200317}, writer:176888, seq_no:0, redo_scn:{val:1713924048588043001, v:0}, end_scn:MAX, trans_version:MIN, status:SET|TRANS|ON_PREPARE|False, ver_next:0x0, linked:0x14b0d6aad750, mds_ctx:0x14b053e0b080, user_data:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}]<->[this:0x14b0d6aad750, ls_id:{id:1001}, tablet_id:{id:1152921504606847581}, writer:176888, seq_no:0, redo_scn:{val:1713924048588043001, v:0}, end_scn:MAX, trans_version:MIN, status:SET|TRANS|ON_PREPARE|False, ver_next:0x0, linked:0x0, mds_ctx:0x14b053e0b080, user_data:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}]<->NULL, total_size:2, obj_to_string(state_):"ON_PREPARE"}}) [2024-04-24 10:00:50.930294] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60379}, multi_version_start=1713895314983386001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.930375] INFO [MDS] create_tablet_on_commit_ (ob_tablet_create_delete_mds_user_data.cpp:163) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=262] create tablet commit(this={tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1713924048823139001, v:0}, create_commit_version:1713924048664122000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}) [2024-04-24 10:00:50.930423] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD296-0-0] [lt=85][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="10.0.0.3:2882", server="10.0.0.2:2882", update_func={old_end_lsn:{lsn:79624410}, new_end_lsn:{lsn:79626650}, old_advance_time_us:1713924048839261, new_ack_time_us:1713924050930416, advance delay(us):2091155}) [2024-04-24 10:00:50.930514] INFO [MDS.EVENT]on_commit_ (mds_node.ipp:239) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=75] ON_COMMIT(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:200317}}, event={alloc:null, timestamp:"2024-04-24 10:00:50.930475", event:"ON_COMMIT", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1713924048823139001, v:0}, create_commit_version:1713924048664122000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176888, seq_no:0, redo_scn:{val:1713924048588043001, v:0}, end_scn:{val:1713924048823139001, v:0}, trans_version:{val:1713924048664122000, v:0}, node_type:1, state:3}) [2024-04-24 10:00:50.930565] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD296-0-0] [lt=91] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=23, this=0x14b0efb296a0, extra_info="palf_id:1") [2024-04-24 10:00:50.930643] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD296-0-0] [lt=65] [PALF STAT LOG LIFE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=636599, this=0x14b0efb298d0, extra_info="palf_id:1") [2024-04-24 10:00:50.930663] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874554][T1004_ApplySrv2][T1004][YB420A000003-000616C2BFC8F285-0-0] [lt=64][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048664122000, v:0}, log_ts:{val:1713924048839739001, v:0}, lsn:{lsn:79625141}, submit_ts:1713924048840938}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=2089699, ctx_lock_wait_time=6) [2024-04-24 10:00:50.930637] INFO [MDS] create_tablet_on_commit_ (ob_tablet_create_delete_mds_user_data.cpp:163) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=95] create tablet commit(this={tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1713924048823139001, v:0}, create_commit_version:1713924048664122000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}) [2024-04-24 10:00:50.930750] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD296-0-0] [lt=95] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=4, avg_log_gen_to_freeze_time=12270, avg_log_gen_to_submit_time=12496, avg_log_submit_to_flush_time=33006, avg_log_submit_to_slide_time=624102) [2024-04-24 10:00:50.930774] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50380}, multi_version_start=1713895315767380000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.930702] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2AE-0-0] [lt=82][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048839739000, v:0}, log_ts:{val:1713924048839739000, v:0}, lsn:{lsn:79624466}, submit_ts:1713924048839739}, this:0x14b055e3afc0, is_inited_:true, trans_id:{txid:176913}, ls_id:{id:1}, ctx:0x14b055e38050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=2090933, ctx_lock_wait_time=8) [2024-04-24 10:00:50.930782] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874554][T1004_ApplySrv2][T1004][YB420A000003-000616C2BFC8F285-0-0] [lt=95] notify MDS(ret=0, trans_id_={txid:176888}, ls_id_={id:1}, notify_type=4, log_ts={val:1713924048839739001, v:0}, notify_array.count()=1, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:true, type:6, data_.length():1313}], total_time=0) [2024-04-24 10:00:50.930820] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD296-0-0] [lt=63][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=304861, log_task={header:{begin_lsn:{lsn:79624410}, end_lsn:{lsn:79626650}, log_id:304861, min_scn:{val:1713924048839739000, v:0}, max_scn:{val:1713924048839739001, v:0}, data_len:2184, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:79624410}, data_checksum:1754254014, accum_checksum:1571808166, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924048839990, freeze_ts:1713924048882162, submit_ts:1713924048882360, flushed_ts:1713924048894789, gen_to_freeze cost time:42172, gen_to_submit cost time:42370, submit_to_flush cost time:12429}, fs_cb_begin_ts=1713924050930522, log_life_time=2090532) [2024-04-24 10:00:50.930820] INFO [MDS.EVENT]on_commit_ (mds_node.ipp:239) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=145] ON_COMMIT(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:1152921504606847581}}, event={alloc:null, timestamp:"2024-04-24 10:00:50.930806", event:"ON_COMMIT", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1713924048823139001, v:0}, create_commit_version:1713924048664122000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176888, seq_no:0, redo_scn:{val:1713924048588043001, v:0}, end_scn:{val:1713924048823139001, v:0}, trans_version:{val:1713924048664122000, v:0}, node_type:1, state:3}) [2024-04-24 10:00:50.930988] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=152] notify MDS(ret=0, trans_id_={txid:176888}, ls_id_={id:1001}, notify_type=4, log_ts={val:1713924048823139001, v:0}, notify_array.count()=1, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], total_time=0) [2024-04-24 10:00:50.931175] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=95] rpc statistics(total_trans_msg_count=7, total_batch_msg_count=6) [2024-04-24 10:00:50.931243] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2AE-0-0] [lt=163] rpc statistics(total_trans_msg_count=7, total_batch_msg_count=6) [2024-04-24 10:00:50.931409] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:233}, multi_version_start=1713895315767380000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.931992] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50114}, multi_version_start=1713895315808716000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.931846] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [874489][T1004_FreInfoRe][T1004][YB420A000003-000616C2B6EA33BF-0-0] [lt=104] slow inner sql(last_ret=0, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1513339) [2024-04-24 10:00:50.932189] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [874706][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=115] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:769, free_obj_hold_cnt:74, allocator used:3351768, allocator total:3466624}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:72, free_obj_hold_cnt:32, allocator used:6821464, allocator total:6821464}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():154752, full allocator used:1600, full allocator total:220288}, tablets_mem=10508376, tablets_mem_limit=257698020, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:23, free_obj_hold_cnt:163, allocator used:369024, allocator total:392448}, pending_cnt=0, wait_gc_count=0, tablet count=841) [2024-04-24 10:00:50.932601] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50192}, multi_version_start=1713895315808716000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.932658] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=81][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.933077] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=98] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60350}, multi_version_start=1713895316011635000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.933229] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [862406][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=97] handle timeout(ret=0, *this={this:0x14b15a1754d0, ref:4, trans_id:{txid:568573}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924078910775, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048911462], ctx_create_time:1713924048910937}{ls_id:{id:1}, session_id:3221542304, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.3:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924048909715, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924048910937000, v:0}, log_ts:{val:1713924048910937000, v:0}, lsn:{lsn:314133219}, submit_ts:1713924048910937}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568573}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:568573}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924048910937000, v:0}, start_scn:{val:1713924048910937000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875429046312 max_table_version=1713875429046312 trans_version={val:1713924048910937000, v:0} commit_version={val:1713924048910937000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:568573} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=2854444523 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924048910937000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048910937, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=400000) [2024-04-24 10:00:50.933409] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [862406][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=170] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1713924048910937000, v:0}, log_ts:{val:1713924048910937000, v:0}, lsn:{lsn:314133219}, submit_ts:1713924048910937}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568573}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924048910937000, v:0}, log_ts:{val:1713924048910937000, v:0}, lsn:{lsn:314133219}, submit_ts:1713924048910937}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568573}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:50.933539] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60259}, multi_version_start=1713895316011635000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.933836] INFO [LIB] stat (utility.h:1197) [2320947][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=301] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=109, this=0x14b0c0e2b470, extra_info="") [2024-04-24 10:00:50.933945] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=119] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.933981] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [2320947][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=118] [PALF STAT GROUP LOG INFO](palf_id=1001, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=3, avg_log_batch_cnt=1, total_group_log_size=1981, avg_group_log_size=660) [2024-04-24 10:00:50.934100] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [2320947][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=96] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, append size=6150) [2024-04-24 10:00:50.934187] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=98] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:188}, multi_version_start=1713895316235730000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.934216] INFO [LIB] stat (utility.h:1197) [2320947][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=87] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x14b0c0f0ae50, extra_info="id:1001") [2024-04-24 10:00:50.934402] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2366) [861400][pnio1][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=62][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:176888}, sender_ls_id={id:1}, receiver_ls_id={id:9223372036854775807}, msg_type=21, status=-4018, receiver_addr="10.0.0.2:2882", request_id=1713924050932585) [2024-04-24 10:00:50.934545] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:285) [861400][pnio1][T0][YB420A000003-000616C2BFC8F2A2-0-0] [lt=136][errcode=0] trx rpc callback(ret=0, status=-4018, dst="10.0.0.2:2882", result={status:-4018, send_timestamp:1713924050932571, private_data:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:50.934838] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=175] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:224}, multi_version_start=1713895316235730001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.935415] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=120] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:308}, multi_version_start=1713895316235730001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.935768] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:426) [874489][T1004_FreInfoRe][T1004][Y0-0000000000000000-0-0] [lt=313] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2024-04-24 10:00:50.936039] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:369}, multi_version_start=1713895316235730001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.936575] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:493}, multi_version_start=1713895316423756000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.936633] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2108) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=95] do_fetch_log_ finished(ret=0, palf_id=1002, self="10.0.0.3:2882", dest="10.0.0.2:2882", fetch_log_count=2048, fetch_start_lsn={lsn:165102580}, prev_lsn={lsn:165102458}, fetch_start_log_id=301037, last_slide_log_id=301036, fetch_log_size=41943040, accepted_mode_pid=5, last_fetch_req_time=1713924050936625, last_fetch_end_lsn={lsn:207045620}, last_fetch_max_log_id=303084, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1002, self:"10.0.0.3:2882", lsn_allocator:{max_log_id:301037, max_lsn:165104153, max_scn:{val:1713924048889699000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:165104153}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:123161113}}, last_submit_lsn:{lsn:165102580}, last_submit_end_lsn:{lsn:165104153}, last_submit_log_id:301037, last_submit_log_pid:5, max_flushed_lsn:{lsn:165102580}, max_flushed_end_lsn:{lsn:165104153}, max_flushed_log_pid:5, committed_end_lsn:{lsn:165102580}, last_slide_log_id:301036, last_slide_scn:{val:1713924048554997000, v:0}, last_slide_lsn:{lsn:165102458}, last_slide_end_lsn:{lsn:165102580}, last_slide_log_pid:5, last_slide_log_accum_checksum:144396744, last_fetch_end_lsn:{lsn:207045620}, last_fetch_max_log_id:303084, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1713924050936625, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x14b13adc2530}) [2024-04-24 10:00:50.937105] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=109] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101011}, multi_version_start=1713895316423756000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.937587] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:284}, multi_version_start=1713895316423756000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.938053] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50149}, multi_version_start=1713895316591099000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.938525] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:307}, multi_version_start=1713895318470989000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.939114] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50240}, multi_version_start=1713895318670997000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.939715] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50355}, multi_version_start=1713895318670997000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.940279] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60457}, multi_version_start=1713895319132417000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.940806] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:182}, multi_version_start=1713895319132417000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.941358] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=136] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101098}, multi_version_start=1713895319340844001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.943158] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [862266][T1002_LSArchive][T1002][YB420A000003-000616C2AE58F0E7-0-0] [lt=266] gc stale ls task succ [2024-04-24 10:00:50.943168] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60387}, multi_version_start=1713895319340844002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.943689] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=130] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60305}, multi_version_start=1713895319906507000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.944191] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=49] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60297}, multi_version_start=1713895320219354000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.944249] INFO [PALF] after_flush_mode_meta (log_mode_mgr.cpp:771) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B43C0573-0-0] [lt=119] after_flush_mode_meta success(ret=0, palf_id=1002, self="10.0.0.3:2882", is_applied_mode_meta=true, mode_meta={version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1713875572331196000, v:0}}, accepted_mode_meta={version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1713875572331196000, v:0}}, applied_mode_meta={version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1713875572331196000, v:0}}) [2024-04-24 10:00:50.944336] INFO [PALF] receive_mode_meta (palf_handle_impl.cpp:2801) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B43C0573-0-0] [lt=77] receive_mode_meta finish(ret=0, ret="OB_SUCCESS", this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, server="10.0.0.2:2882", proposal_id=5, is_applied_mode_meta=true, mode_meta={version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1713875572331196000, v:0}}) [2024-04-24 10:00:50.944422] INFO [PALF] handle_request (log_request_handler.cpp:335) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B43C0573-0-0] [lt=73] receive_mode_meta success(ret=0, palf_id=1002, server="10.0.0.2:2882", req={msg_proposal_id:5, meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1713875572331196000, v:0}}, is_applied_mode_meta:true}, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"10.0.0.3:2882", log_dir:"/data/admin/test/oceanbase/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):5529, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):5529, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1049, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:0}}) [2024-04-24 10:00:50.944656] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60342}, multi_version_start=1713895320985207001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.944963] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F79-0-0] [lt=62][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="10.0.0.3:2882", server="10.0.0.2:2882", update_func={old_end_lsn:{lsn:298522329}, new_end_lsn:{lsn:298523302}, old_advance_time_us:1713924048910239, new_ack_time_us:1713924050944955, advance delay(us):2034716}) [2024-04-24 10:00:50.945009] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_ls_info.cpp:54) [861725][T1_L5_G0][T1][YB420A000003-000616C2A8E93205-0-0] [lt=88][errcode=-4008] execute fail(ret=-4008) [2024-04-24 10:00:50.945077] INFO [LIB] stat (utility.h:1197) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F79-0-0] [lt=52] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=28, this=0x14b0f0aa16a0, extra_info="palf_id:1") [2024-04-24 10:00:50.945168] INFO [LIB] stat (utility.h:1197) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F79-0-0] [lt=76] [PALF STAT LOG LIFE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=789677, this=0x14b0f0aa18d0, extra_info="palf_id:1") [2024-04-24 10:00:50.945208] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.945302] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F79-0-0] [lt=100] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=3, avg_log_gen_to_freeze_time=49726, avg_log_gen_to_submit_time=49973, avg_log_submit_to_flush_time=657480, avg_log_submit_to_slide_time=739703) [2024-04-24 10:00:50.945418] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F79-0-0] [lt=109][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=482555, log_task={header:{begin_lsn:{lsn:298522329}, end_lsn:{lsn:298523302}, log_id:482555, min_scn:{val:1713924048912363000, v:0}, max_scn:{val:1713924048912363000, v:0}, data_len:917, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:298522329}, data_checksum:2494196878, accum_checksum:215992293, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924048913041, freeze_ts:1713924048913057, submit_ts:1713924048913217, flushed_ts:1713924050716350, gen_to_freeze cost time:16, gen_to_submit cost time:176, submit_to_flush cost time:1803133}, fs_cb_begin_ts=1713924050945028, log_life_time=2031987) [2024-04-24 10:00:50.945385] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=161][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048912363000, v:0}, log_ts:{val:1713924048912363000, v:0}, lsn:{lsn:298522385}, submit_ts:1713924048912363}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:338713}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=2033003, ctx_lock_wait_time=10) [2024-04-24 10:00:50.945966] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=123] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101036}, multi_version_start=1713895321186148000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.945874] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=199] rpc statistics(total_trans_msg_count=3, total_batch_msg_count=0) [2024-04-24 10:00:50.946122] WDIAG [CLOG] statistics_cb_cost_ (ob_log_apply_service.cpp:933) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=185][errcode=-4389] cb cost too much time(lsn={lsn:298522385}, scn={val:1713924048912363000, v:0}, idx=24, total_cost_time=2031761, append_cost_time=227, cb_wait_thread_time=2032106, cb_wait_commit_time=0, cb_cost_time=-572, append_start_time=1713924048913027, append_finish_time=1713924048913254, cb_first_handle_time=1713924050945360, cb_finish_time=1713924050944788) [2024-04-24 10:00:50.946300] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB901-0-0] [lt=150][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="10.0.0.3:2882", server="10.0.0.2:2882", update_func={old_end_lsn:{lsn:314133163}, new_end_lsn:{lsn:314133705}, old_advance_time_us:1713924048845761, new_ack_time_us:1713924050946282, advance delay(us):2100521}) [2024-04-24 10:00:50.946431] INFO [LIB] stat (utility.h:1197) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB901-0-0] [lt=64] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=23, this=0x14b14c9b36a0, extra_info="palf_id:1") [2024-04-24 10:00:50.946465] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60005}, multi_version_start=1713895321265127000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.946522] INFO [LIB] stat (utility.h:1197) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB901-0-0] [lt=77] [PALF STAT LOG LIFE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=756016, this=0x14b14c9b38d0, extra_info="palf_id:1") [2024-04-24 10:00:50.946634] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB901-0-0] [lt=97] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=3, avg_log_gen_to_freeze_time=10, avg_log_gen_to_submit_time=210, avg_log_submit_to_flush_time=618765, avg_log_submit_to_slide_time=755806) [2024-04-24 10:00:50.946714] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB901-0-0] [lt=73][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=505464, log_task={header:{begin_lsn:{lsn:314133163}, end_lsn:{lsn:314133705}, log_id:505464, min_scn:{val:1713924048910937000, v:0}, max_scn:{val:1713924048910937000, v:0}, data_len:486, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:314133163}, data_checksum:474681474, accum_checksum:2907004606, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924048911568, freeze_ts:1713924048911580, submit_ts:1713924048911757, flushed_ts:1713924050715474, gen_to_freeze cost time:12, gen_to_submit cost time:189, submit_to_flush cost time:1803717}, fs_cb_begin_ts=1713924050946380, log_life_time=2034812) [2024-04-24 10:00:50.947075] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=119] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60130}, multi_version_start=1713895324302603000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.947157] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=216][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048910937000, v:0}, log_ts:{val:1713924048910937000, v:0}, lsn:{lsn:314133219}, submit_ts:1713924048910937}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568573}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=2036205, ctx_lock_wait_time=9) [2024-04-24 10:00:50.947613] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101064}, multi_version_start=1713895324506362001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.947843] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [862260][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=155] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1713924048276279000, v:0}, new_max_version={val:1713924048276279000, v:0}, last_min_version={val:1713924048246903000, v:0}, last_max_version={val:1713924048246903000, v:0}, total_time_us=2038693, delta_us=2671557) [2024-04-24 10:00:50.948024] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [862260][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=171] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1713924048276279000, v:0}, min_version={val:1713924048276279000, v:0}, max_version={val:1713924048276279000, v:0}, in_service=true, leader_epoch=5) [2024-04-24 10:00:50.948213] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:404}, multi_version_start=1713895324506362001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.948553] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4087) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C49-0-0] [lt=69][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1001, self="10.0.0.3:2882", server="10.0.0.2:2882", update_func={old_end_lsn:{lsn:1002196895}, new_end_lsn:{lsn:1002197017}, old_advance_time_us:1713924048883940, new_ack_time_us:1713924050948544, advance delay(us):2064604}) [2024-04-24 10:00:50.948770] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50393}, multi_version_start=1713895324506362001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.948784] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C49-0-0] [lt=77] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=53, this=0x14b1336eb6a0, extra_info="palf_id:1001") [2024-04-24 10:00:50.948929] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C49-0-0] [lt=80] [PALF STAT LOG LIFE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=687561, this=0x14b1336eb8d0, extra_info="palf_id:1001") [2024-04-24 10:00:50.949005] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C49-0-0] [lt=69] [PALF STAT LOG TASK TIME](palf_id=1001, self="10.0.0.3:2882", total_slide_log_cnt=3, avg_log_gen_to_freeze_time=12239, avg_log_gen_to_submit_time=12431, avg_log_submit_to_flush_time=596486, avg_log_submit_to_slide_time=675130) [2024-04-24 10:00:50.949144] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C49-0-0] [lt=122][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=673424, log_task={header:{begin_lsn:{lsn:1002196895}, end_lsn:{lsn:1002197017}, log_id:673424, min_scn:{val:1713924048914830000, v:0}, max_scn:{val:1713924048914830000, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1002196895}, data_checksum:2222604892, accum_checksum:817730249, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924049103909, freeze_ts:1713924049103926, submit_ts:1713924049104061, flushed_ts:1713924050714622, gen_to_freeze cost time:17, gen_to_submit cost time:152, submit_to_flush cost time:1610561}, fs_cb_begin_ts=1713924050948719, log_life_time=1844810) [2024-04-24 10:00:50.949383] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101080}, multi_version_start=1713895325103982000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.949359] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD297-0-0] [lt=132] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=6, this=0x14b0c0ef7470, extra_info="") [2024-04-24 10:00:50.949346] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2108) [861573][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=182] do_fetch_log_ finished(ret=0, palf_id=1, self="10.0.0.3:2882", dest="10.0.0.2:2882", fetch_log_count=2048, fetch_start_lsn={lsn:179748114}, prev_lsn={lsn:179747121}, fetch_start_log_id=364245, last_slide_log_id=364244, fetch_log_size=41943040, accepted_mode_pid=5, last_fetch_req_time=1713924050949340, last_fetch_end_lsn={lsn:221691154}, last_fetch_max_log_id=366292, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1, self:"10.0.0.3:2882", lsn_allocator:{max_log_id:364245, max_lsn:179748652, max_scn:{val:1713924048988077000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:179748652}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:137805612}}, last_submit_lsn:{lsn:179748114}, last_submit_end_lsn:{lsn:179748652}, last_submit_log_id:364245, last_submit_log_pid:5, max_flushed_lsn:{lsn:179748114}, max_flushed_end_lsn:{lsn:179748652}, max_flushed_log_pid:5, committed_end_lsn:{lsn:179748114}, last_slide_log_id:364244, last_slide_scn:{val:1713924048840067000, v:0}, last_slide_lsn:{lsn:179747121}, last_slide_end_lsn:{lsn:179748114}, last_slide_log_pid:5, last_slide_log_accum_checksum:870161191, last_fetch_end_lsn:{lsn:221691154}, last_fetch_max_log_id:366292, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1713924050949340, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x14b186b46530}) [2024-04-24 10:00:50.949483] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD297-0-0] [lt=87] [PALF STAT GROUP LOG INFO](palf_id=1002, self="10.0.0.3:2882", role="FOLLOWER", total_group_log_cnt=1, avg_log_batch_cnt=0, total_group_log_size=2214, avg_group_log_size=2214) [2024-04-24 10:00:50.949575] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD297-0-0] [lt=84] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="10.0.0.3:2882", committed size=122) [2024-04-24 10:00:50.949901] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD297-0-0] [lt=62] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=130, this=0x14b0c0ef76a0, extra_info="palf_id:1002") [2024-04-24 10:00:50.949993] INFO eloop_run (eloop.c:143) [861404][pnio2][T0][Y0-0000000000000000-0-0] [lt=99] PNIO [ratelimit] time: 1713924050949985, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007786, add_bytes: 0 [2024-04-24 10:00:50.950006] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD297-0-0] [lt=92] [PALF STAT LOG LIFE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=2047455, this=0x14b0c0ef78d0, extra_info="palf_id:1002") [2024-04-24 10:00:50.950062] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50424}, multi_version_start=1713895325103982000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.950047] INFO [PALF] read (log_cache.cpp:100) [874559][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=302] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=122, hit_cnt=1, read_cnt=1, hit rate=1.000000000000000000e+00) [2024-04-24 10:00:50.950182] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD297-0-0] [lt=143] [PALF STAT LOG TASK TIME](palf_id=1002, self="10.0.0.3:2882", total_slide_log_cnt=1, avg_log_gen_to_freeze_time=12, avg_log_gen_to_submit_time=190, avg_log_submit_to_flush_time=1809138, avg_log_submit_to_slide_time=2047265) [2024-04-24 10:00:50.950264] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD297-0-0] [lt=69][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=299917, log_task={header:{begin_lsn:{lsn:40358611}, end_lsn:{lsn:40358733}, log_id:299917, min_scn:{val:1713924048832745000, v:0}, max_scn:{val:1713924048832745000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:40358489}, prev_proposal_id:5, committed_end_lsn:{lsn:40358611}, data_checksum:1715074492, accum_checksum:1700048438, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924048902254, freeze_ts:1713924048902266, submit_ts:1713924048902444, flushed_ts:1713924050711582, gen_to_freeze cost time:12, gen_to_submit cost time:190, submit_to_flush cost time:1809138}, fs_cb_begin_ts=1713924050949709, log_life_time=2047455) [2024-04-24 10:00:50.950582] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=119] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.950623] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=128] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:197}, multi_version_start=1713895325103982000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.950813] INFO [LIB] stat (utility.h:1197) [874562][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=71] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=568, this=0x14b1312048f8, extra_info="") [2024-04-24 10:00:50.950927] INFO [LIB] stat (utility.h:1197) [874562][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=98] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=27, this=0x14b131204b28, extra_info="") [2024-04-24 10:00:50.951183] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50381}, multi_version_start=1713895325103982004, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.951740] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:249}, multi_version_start=1713895325276947002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.952289] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:103}, multi_version_start=1713895325276947002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.952785] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50373}, multi_version_start=1713895325478504000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.953362] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=144] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50437}, multi_version_start=1713895325478504000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.953734] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=60][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:314133705}], scn_array=[{val:1713924048929552000, v:0}], curr_size=608, accum_size=608, time_cost=237638) [2024-04-24 10:00:50.954068] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101083}, multi_version_start=1713895325952788000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.954670] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101062}, multi_version_start=1713895325952788002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.955055] INFO [PALF] try_reset_last_fetch_log_info_ (log_sliding_window.cpp:1838) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C4A-0-0] [lt=78] reset last fetch log info(palf_id=1002, self="10.0.0.3:2882", expected_end_lsn={lsn:207045620}) [2024-04-24 10:00:50.955152] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60263}, multi_version_start=1713895325952788002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.955209] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C4A-0-0] [lt=89] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=8, this=0x14b13adc3470, extra_info="") [2024-04-24 10:00:50.955387] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C4A-0-0] [lt=147] [PALF STAT GROUP LOG INFO](palf_id=1002, self="10.0.0.3:2882", role="FOLLOWER", total_group_log_cnt=1, avg_log_batch_cnt=0, total_group_log_size=122, avg_group_log_size=122) [2024-04-24 10:00:50.955433] INFO [PALF] after_flush_mode_meta (log_mode_mgr.cpp:771) [2325892][T1_L0_G1][T1][YB420A000002-000616C2A9CC0583-0-0] [lt=91] after_flush_mode_meta success(ret=0, palf_id=1, self="10.0.0.3:2882", is_applied_mode_meta=true, mode_meta={version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta={version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, applied_mode_meta={version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-24 10:00:50.955473] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C4A-0-0] [lt=78] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="10.0.0.3:2882", committed size=1695) [2024-04-24 10:00:50.955536] INFO [PALF] receive_mode_meta (palf_handle_impl.cpp:2801) [2325892][T1_L0_G1][T1][YB420A000002-000616C2A9CC0583-0-0] [lt=90] receive_mode_meta finish(ret=0, ret="OB_SUCCESS", this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, server="10.0.0.2:2882", proposal_id=5, is_applied_mode_meta=true, mode_meta={version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-24 10:00:50.955734] INFO [PALF] handle_request (log_request_handler.cpp:335) [2325892][T1_L0_G1][T1][YB420A000002-000616C2A9CC0583-0-0] [lt=190] receive_mode_meta success(ret=0, palf_id=1, server="10.0.0.2:2882", req={msg_proposal_id:5, meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, is_applied_mode_meta:true}, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"10.0.0.3:2882", log_dir:"/data/admin/test/oceanbase/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):171, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-04-24 10:00:50.955742] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C4A-0-0] [lt=166] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=56, this=0x14b13adc36a0, extra_info="palf_id:1002") [2024-04-24 10:00:50.955784] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:152}, multi_version_start=1713895326155077000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.955850] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C4A-0-0] [lt=96] [PALF STAT LOG LIFE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=1097580, this=0x14b13adc38d0, extra_info="palf_id:1002") [2024-04-24 10:00:50.955934] INFO [PALF] read (log_cache.cpp:100) [862236][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=147] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1695, hit_cnt=2, read_cnt=2, hit rate=1.000000000000000000e+00) [2024-04-24 10:00:50.955976] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C4A-0-0] [lt=119] [PALF STAT LOG TASK TIME](palf_id=1002, self="10.0.0.3:2882", total_slide_log_cnt=2, avg_log_gen_to_freeze_time=34, avg_log_gen_to_submit_time=162, avg_log_submit_to_flush_time=65824, avg_log_submit_to_slide_time=1097418) [2024-04-24 10:00:50.956045] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C4A-0-0] [lt=63][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=301037, log_task={header:{begin_lsn:{lsn:165102580}, end_lsn:{lsn:165104153}, log_id:301037, min_scn:{val:1713924048889699000, v:0}, max_scn:{val:1713924048889699000, v:0}, data_len:1517, proposal_id:5, prev_lsn:{lsn:165102458}, prev_proposal_id:5, committed_end_lsn:{lsn:165102580}, data_checksum:1526117220, accum_checksum:1187085275, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924048891857, freeze_ts:1713924048891917, submit_ts:1713924048892132, flushed_ts:1713924048896821, gen_to_freeze cost time:60, gen_to_submit cost time:275, submit_to_flush cost time:4689}, fs_cb_begin_ts=1713924050955687, log_life_time=2063830) [2024-04-24 10:00:50.956290] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50364}, multi_version_start=1713895326155077000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.956366] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2ABEC0CED-0-0] [lt=112] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924037247474, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924050955440, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:50.956546] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [874422][T1004_TeRec][T1003][YB420A000003-000616C2B5596E96-0-0] [lt=80] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=2957) [2024-04-24 10:00:50.956673] INFO [LIB] stat (utility.h:1197) [874422][T1004_TeRec][T1003][YB420A000003-000616C2B5596E96-0-0] [lt=97] [PALF STAT APPEND COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=0, this=0x14b0f0a74e50, extra_info="id:1") [2024-04-24 10:00:50.956772] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50279}, multi_version_start=1713895326399630000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.956950] INFO [LIB] stat (utility.h:1197) [862286][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=104] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=125, this=0x14b16c6048f8, extra_info="") [2024-04-24 10:00:50.957045] INFO [LIB] stat (utility.h:1197) [862286][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=79] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=413, this=0x14b16c604b28, extra_info="") [2024-04-24 10:00:50.957322] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=135] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60167}, multi_version_start=1713895326399630000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.957851] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=122] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101018}, multi_version_start=1713895326486462001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.958548] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=133] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60424}, multi_version_start=1713895326591589000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.959053] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1073) [874502][T1004_ReplayPro][T1004][Y0-0000000000000000-0-0] [lt=140] replay status is not follower(min_unreplayed_lsn={lsn:69192977}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713921656897811, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:69192977}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:69192977}, end_lsn:{lsn:69192977}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x14b0efb2f470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-04-24 10:00:50.959159] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60438}, multi_version_start=1713895326591589000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.959166] INFO [CLOG] operator() (ob_log_replay_service.cpp:1590) [874502][T1004_ReplayPro][T1004][Y0-0000000000000000-0-0] [lt=106] get_replay_process success(id={id:1}, replayed_log_size=69192977, unreplayed_log_size=0) [2024-04-24 10:00:50.959256] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1073) [874502][T1004_ReplayPro][T1004][Y0-0000000000000000-0-0] [lt=81] replay status is not follower(min_unreplayed_lsn={lsn:82126713}, base_lsn={lsn:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713921656936517, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:82126713}, next_to_submit_scn_:{val:1713900501167593003, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1713877500257917000, v:0}, iterator_:{iterator_impl:{buf_:0x14b12fd69000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:82126713}, end_lsn:{lsn:82126713}, read_buf:{buf_len_:8192, buf_:0x14b12fd69000}, block_size:67104768, log_storage_:0x14b0c0e31470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-04-24 10:00:50.959363] INFO [CLOG] operator() (ob_log_replay_service.cpp:1590) [874502][T1004_ReplayPro][T1004][Y0-0000000000000000-0-0] [lt=101] get_replay_process success(id={id:1001}, replayed_log_size=82126713, unreplayed_log_size=0) [2024-04-24 10:00:50.959482] INFO [CLOG] operator() (ob_log_replay_service.cpp:1590) [874502][T1004_ReplayPro][T1004][Y0-0000000000000000-0-0] [lt=103] get_replay_process success(id={id:1002}, replayed_log_size=40358733, unreplayed_log_size=0) [2024-04-24 10:00:50.959590] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:159) [874502][T1004_ReplayPro][T1004][Y0-0000000000000000-0-0] [lt=90] dump tenant replay process(tenant_id=1004, unreplayed_log_size(MB)=0, estimate_time(second)=0, replayed_log_size(MB)=182, last_replayed_log_size(MB)=182, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2024-04-24 10:00:50.959723] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60440}, multi_version_start=1713895326591589000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.960216] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101034}, multi_version_start=1713895326798729000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.960705] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [862534][T1002_L0_G5][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=81] is logging [2024-04-24 10:00:50.960751] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=98] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:255}, multi_version_start=1713895326798729000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.960862] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [862534][T1002_L0_G5][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=132] submit log with lock(ret=-4023, service_type_=0, last_id=1713924066491232000, limited_id=1713924066491232000) [2024-04-24 10:00:50.961339] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101084}, multi_version_start=1713895326920409000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.961980] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=104] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60233}, multi_version_start=1713895326920409000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.962463] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50142}, multi_version_start=1713895327204375000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.963045] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50120}, multi_version_start=1713895327204375000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.963286] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [862534][T1002_L0_G5][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=112] is logging [2024-04-24 10:00:50.963422] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [862534][T1002_L0_G5][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=111] submit log with lock(ret=-4023, service_type_=0, last_id=1713924066491232000, limited_id=1713924066491232000) [2024-04-24 10:00:50.963597] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=98][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67159461}], scn_array=[{val:1713924049917601001, v:0}], curr_size=128, accum_size=128, time_cost=247138) [2024-04-24 10:00:50.963657] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50406}, multi_version_start=1713895327204375000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.963821] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=62][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:40358733}], scn_array=[{val:1713924049250797000, v:0}], curr_size=2214, accum_size=2214, time_cost=14221) [2024-04-24 10:00:50.963961] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=101][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:298523302}], scn_array=[{val:1713924049017564000, v:0}], curr_size=1094, accum_size=1094, time_cost=246912) [2024-04-24 10:00:50.963968] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=50][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:79626772}], scn_array=[{val:1713924050897126000, v:0}], curr_size=498, accum_size=498, time_cost=64963) [2024-04-24 10:00:50.964107] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=86][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:88123732}], scn_array=[{val:1713924048914830001, v:0}], curr_size=157, accum_size=157, time_cost=29951) [2024-04-24 10:00:50.964389] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=123] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50315}, multi_version_start=1713895327402272000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.965199] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=121] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101068}, multi_version_start=1713895327402272000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.965841] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=159] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:439}, multi_version_start=1713895327402272000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.966400] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=128] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:165}, multi_version_start=1713895327607056000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.966530] INFO [PALF] get_log (palf_handle_impl.cpp:4002) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B39C6514-0-0] [lt=81] submit fetch log task success(ret=0, palf_id=1, server="10.0.0.2:2882", fetch_type=0, prev_lsn={lsn:67159217}, start_lsn={lsn:67159339}, fetch_log_size=41943040, fetch_log_count=2048, accepted_mode_pid=5) [2024-04-24 10:00:50.966601] INFO [PALF] handle (fetch_log_engine.cpp:284) [862148][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=122] handle fetch_log_task(fetch_log_task={timestamp_us:1713924050966478, id:1, server:"10.0.0.2:2882", fetch_type:"FETCH_LOG_FOLLOWER", proposal_id:5, prev_lsn:{lsn:67159217}, start_lsn:{lsn:67159339}, log_size:41943040, log_count:2048, accepted_mode_pid:5}) [2024-04-24 10:00:50.966837] INFO [PALF] read (log_cache.cpp:100) [862148][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=155] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=372, hit_cnt=2, read_cnt=2, hit rate=1.000000000000000000e+00) [2024-04-24 10:00:50.966979] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=107][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:50.967057] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:157}, multi_version_start=1713895327637212001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.967078] WDIAG [PALF] batch_fetch_log_each_round_ (palf_handle_impl.cpp:3776) [862148][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=149][errcode=-4008] iterator next failed(ret=-4008, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, iterator={iterator_impl:{buf_:0x14b0efcb9000, next_round_pread_size:3678208, curr_read_pos:250, curr_read_buf_start_pos:0, curr_read_buf_end_pos:250, log_storage_:{IteratorStorage:{start_lsn:{lsn:67159339}, end_lsn:{lsn:67159589}, read_buf:{buf_len_:8192, buf_:0x14b0efcb9000}, block_size:67104768, log_storage_:0x14b1470b1470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713924049917601001, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:1, group_size:72, proposal_id:5, committed_lsn:{lsn:67159339}, max_scn:{val:1713924049917601001, v:0}, accumulated_checksum:2685460241, log_id:259319, flag:1}}, init_mode_version:0, accumulate_checksum:2685460241, curr_entry_is_padding:0, padding_entry_size:72, padding_entry_scn:{val:1713924049917601001, v:0}}}, replayable_point={val:1713924044057788000, v:0}, prev_lsn={lsn:67159461}) [2024-04-24 10:00:50.967270] INFO [PALF] fetch_log_from_storage_ (palf_handle_impl.cpp:3713) [862148][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=109] fetch_log_from_storage_ finished(ret=0, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, prev_lsn={lsn:67159217}, iterator={iterator_impl:{buf_:0x14b0efcb9000, next_round_pread_size:3678208, curr_read_pos:250, curr_read_buf_start_pos:0, curr_read_buf_end_pos:250, log_storage_:{IteratorStorage:{start_lsn:{lsn:67159339}, end_lsn:{lsn:67159589}, read_buf:{buf_len_:8192, buf_:0x14b0efcb9000}, block_size:67104768, log_storage_:0x14b1470b1470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713924049917601001, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:1, group_size:72, proposal_id:5, committed_lsn:{lsn:67159339}, max_scn:{val:1713924049917601001, v:0}, accumulated_checksum:2685460241, log_id:259319, flag:1}}, init_mode_version:0, accumulate_checksum:2685460241, curr_entry_is_padding:0, padding_entry_size:72, padding_entry_scn:{val:1713924049917601001, v:0}}}, fetch_log_count=2048, has_fetched_log_count=2) [2024-04-24 10:00:50.967436] INFO [PALF] submit_fetch_mode_meta_resp (log_mode_mgr.cpp:826) [862148][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=105] submit_change_mode_meta_req success(ret=0, palf_id=1, server="10.0.0.2:2882", mode_meta={version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_pid=5) [2024-04-24 10:00:50.967514] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60145}, multi_version_start=1713895327637212001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.967523] INFO [PALF] fetch_log_from_storage (palf_handle_impl.cpp:3498) [862148][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=78] [PALF STAT FETCH LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=41943040) [2024-04-24 10:00:50.967636] INFO [LIB] stat (utility.h:1197) [862148][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=92] [PALF STAT FETCH LOG TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=111, this=0x14b1679f2600, extra_info="") [2024-04-24 10:00:50.967729] INFO [LIB] stat (utility.h:1197) [862148][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=76] [PALF STAT FETCH LOG EXECUTE COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1033, this=0x14b1679f2830, extra_info="") [2024-04-24 10:00:50.968379] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:363}, multi_version_start=1713895327637212001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.969020] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60444}, multi_version_start=1713895327805984000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.969505] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50329}, multi_version_start=1713895327863136000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.970196] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60226}, multi_version_start=1713895328006580000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.970559] INFO [LIB] stat (utility.h:1197) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=84] [PALF STAT WRITE LOG COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=9705, this=0x14b13adcd3b0, extra_info="palf_id:1002") [2024-04-24 10:00:50.970753] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=186][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:165104153}], scn_array=[{val:1713924048889699001, v:0}], curr_size=122, accum_size=1695, time_cost=15082) [2024-04-24 10:00:50.970777] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:225}, multi_version_start=1713895328060090000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.970849] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=89] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=1695) [2024-04-24 10:00:50.971061] INFO [LIB] stat (utility.h:1214) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=149] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=113, this=0x14b1679f73b0, extra_info="") [2024-04-24 10:00:50.971225] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=99] [PALF STAT FLUSH CB COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=129, this=0x14b13adcd5e0, extra_info="palf_id:1002") [2024-04-24 10:00:50.971374] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60441}, multi_version_start=1713895328060090000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.971961] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60253}, multi_version_start=1713895328060090000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.972421] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101073}, multi_version_start=1713895328273183000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.973115] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60383}, multi_version_start=1713895328273183000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.973622] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:427}, multi_version_start=1713895328431104000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.974147] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:294}, multi_version_start=1713895328469974000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.974564] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.975040] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=52] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:392}, multi_version_start=1713895328540808000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.975561] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50379}, multi_version_start=1713895328540808000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.976138] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=77][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:79627270}], scn_array=[{val:1713924050897126001, v:0}], curr_size=157, accum_size=655, time_cost=11398) [2024-04-24 10:00:50.976314] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=76][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:298524396}], scn_array=[{val:1713924050955440000, v:0}], curr_size=1015, accum_size=2109, time_cost=11565) [2024-04-24 10:00:50.976344] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=151][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:88123889}], scn_array=[{val:1713924050938294000, v:0}], curr_size=3298, accum_size=3455, time_cost=11176) [2024-04-24 10:00:50.976350] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=126] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50450}, multi_version_start=1713895328540808000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.976858] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:196}, multi_version_start=1713895328834089000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.977594] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=145] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60243}, multi_version_start=1713895328834089000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.978203] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=111] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:349}, multi_version_start=1713895328988021000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.978750] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60166}, multi_version_start=1713895328988021000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.979348] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50334}, multi_version_start=1713895328988021000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.979941] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101031}, multi_version_start=1713895329281243000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.979941] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=135] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:50.980046] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=70] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:50.980151] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=98] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:50.980517] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60459}, multi_version_start=1713895329281243000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.981141] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=98] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60425}, multi_version_start=1713895329382281000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.981159] INFO [STORAGE.TRANS] handle_tx_commit_timeout (ob_trans_service_v4.cpp:393) [861656][T1_TransTimeWhe][T1][Y0-0000000000000000-0-0] [lt=94] handle tx commit timeout(ret=0, tx_id={txid:169795}, ref_cnt=4, cb_executed=false) [2024-04-24 10:00:50.981686] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50408}, multi_version_start=1713895329494196001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.982236] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60349}, multi_version_start=1713895329494196001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.982690] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60113}, multi_version_start=1713895329673646000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.983229] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60352}, multi_version_start=1713895329673646000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.983982] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60180}, multi_version_start=1713895329673646000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.984418] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:473}, multi_version_start=1713895329886255000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.985038] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50325}, multi_version_start=1713895329886255001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.985450] INFO [STORAGE.TRANS] handle_tx_commit_result_ (ob_trans_service_v4.cpp:549) [861400][pnio1][T1][YB420A000003-000616C2ACD902D5-0-0] [lt=66] handle tx commit result(ret=0, ref_cnt_0=4, tx={this:0x14b186574fe0, tx_id:{txid:169795}, state:5, addr:"10.0.0.3:2882", tenant_id:1, session_id:3221540160, assoc_session_id:3221540160, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924049402283, op_sn:4, alloc_ts:1713924049403545, active_ts:1713924049403545, commit_ts:1713924049409760, finish_ts:-1, timeout_us:86400000000, lock_timeout_us:-1, expire_ts:1714010449403545, coord_id:{id:1}, parts:[{id:{id:1}, addr:"10.0.0.2:2882", epoch:1430291097262351, first_scn:1713924049406085, last_scn:1713924049406085, last_touch_ts:1}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:8, commit_cb:0x14b1187d4988, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:1713924059410543, commit_task_.is_registered():true, ref:4}, commit_fin=false, result=-4023) [2024-04-24 10:00:50.985505] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60431}, multi_version_start=1713895330086669000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.985607] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2366) [861400][pnio1][T1][YB420A000003-000616C2ACD902D5-0-0] [lt=150][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:169795}, sender_ls_id={id:9223372036854775807}, receiver_ls_id={id:1}, msg_type=20, status=-4023, receiver_addr="10.0.0.2:2882", request_id=4) [2024-04-24 10:00:50.985745] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:285) [861400][pnio1][T0][YB420A000003-000616C2ACD902D5-0-0] [lt=132][errcode=0] trx rpc callback(ret=0, status=-4023, dst="10.0.0.2:2882", result={status:-4023, send_timestamp:1713924050981000, private_data:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:50.986093] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50292}, multi_version_start=1713895330248295000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.986739] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60410}, multi_version_start=1713895330248295000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.987325] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:372}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.988070] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60491}, multi_version_start=1713895330531386003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.988698] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1713895330531386003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.989310] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60165}, multi_version_start=1713895330531386003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.989310] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [862324][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=271] LogLoopThread round_cost_time(us)(round_cost_time=34) [2024-04-24 10:00:50.989980] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101005}, multi_version_start=1713895330943452001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.990537] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:362}, multi_version_start=1713895331182976000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.991197] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:405}, multi_version_start=1713895331182976000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.991768] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101027}, multi_version_start=1713895331388911001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.992354] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:210}, multi_version_start=1713895331388911001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.992538] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [874646][T1003_PlanCache][T1003][Y0-0000000000000000-0-0] [lt=61] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1003) [2024-04-24 10:00:50.992663] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [874646][T1003_PlanCache][T1003][Y0-0000000000000000-0-0] [lt=99] start lib cache evict(tenant_id=1003, mem_hold=54525952, mem_limit=64424505, cache_obj_num=170, cache_node_num=168) [2024-04-24 10:00:50.992771] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [874646][T1003_PlanCache][T1003][Y0-0000000000000000-0-0] [lt=98] end lib cache evict(tenant_id=1003, cache_evict_num=0, mem_hold=54525952, mem_limit=64424505, cache_obj_num=170, cache_node_num=168) [2024-04-24 10:00:50.992931] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [874646][T1003_PlanCache][T1003][Y0-0000000000000000-0-0] [lt=137] schedule next cache evict task(evict_interval=5000000) [2024-04-24 10:00:50.993032] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=109] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60103}, multi_version_start=1713895332068660001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.993534] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.994153] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50246}, multi_version_start=1713895332068660001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.994689] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50251}, multi_version_start=1713895332272611000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.995193] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60262}, multi_version_start=1713895332272611001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.995667] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50003}, multi_version_start=1713895332272611001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.996536] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=104] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60279}, multi_version_start=1713895332472335000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.996457] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [874646][T1003_PlanCache][T1003][Y0-0000000000000000-0-0] [lt=90] Dumping All Cache Objs(alloc_obj_list.count()=170, alloc_obj_list=[{obj_id:291, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179593}, {obj_id:1370, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196868}, {obj_id:5287, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178961}, {obj_id:5289, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181217}, {obj_id:5322, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175250}, {obj_id:5384, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198516}, {obj_id:5412, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181020}, {obj_id:5966, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194797}, {obj_id:5967, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189174}, {obj_id:5969, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191942}, {obj_id:5975, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200417}, {obj_id:5976, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186066}, {obj_id:5978, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188048}, {obj_id:6005, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:177974}, {obj_id:6619, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:4, added_to_lc:true, mem_used:191152}, {obj_id:6623, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188982}, {obj_id:6628, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:6634, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184112}, {obj_id:6635, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191482}, {obj_id:6887, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172112}, {obj_id:7105, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172528}, {obj_id:7106, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175256}, {obj_id:7297, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:227196}, {obj_id:7298, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7313, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197500}, {obj_id:7314, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185946}, {obj_id:7371, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7373, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7377, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7398, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188403}, {obj_id:7399, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7406, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183681}, {obj_id:7407, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186090}, {obj_id:7408, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7409, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190249}, {obj_id:7410, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198112}, {obj_id:7411, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183906}, {obj_id:7412, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193848}, {obj_id:7414, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186294}, {obj_id:7415, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190783}, {obj_id:7416, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7417, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184692}, {obj_id:7418, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195616}, {obj_id:7420, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173949}, {obj_id:7421, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188297}, {obj_id:7424, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175846}, {obj_id:7426, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182028}, {obj_id:7436, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7438, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7439, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7442, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173104}, {obj_id:7443, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177587}, {obj_id:7444, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:231826}, {obj_id:7445, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7446, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184351}, {obj_id:7447, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187634}, {obj_id:7448, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7449, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239808}, {obj_id:7450, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217216}, {obj_id:7451, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7452, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7453, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7454, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174320}, {obj_id:7455, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175106}, {obj_id:7456, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175786}, {obj_id:7457, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181421}, {obj_id:7458, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173408}, {obj_id:7459, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176132}, {obj_id:7460, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174048}, {obj_id:7461, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189428}, {obj_id:7462, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188653}, {obj_id:7463, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174304}, {obj_id:7464, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217516}, {obj_id:7465, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215079}, {obj_id:7466, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178561}, {obj_id:7467, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174912}, {obj_id:7468, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7469, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179603}, {obj_id:7470, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172688}, {obj_id:7471, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186978}, {obj_id:7472, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194259}, {obj_id:7473, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179903}, {obj_id:7474, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193660}, {obj_id:7475, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180401}, {obj_id:7476, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7477, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173504}, {obj_id:7478, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7479, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177273}, {obj_id:7480, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7481, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237328}, {obj_id:7482, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7483, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216328}, {obj_id:7484, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7485, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7486, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185840}, {obj_id:7487, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183952}, {obj_id:7488, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172736}, {obj_id:7489, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7490, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186515}, {obj_id:7491, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178816}, {obj_id:7492, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185947}, {obj_id:7493, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179880}, {obj_id:7494, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183760}, {obj_id:7495, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172960}, {obj_id:7496, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173648}, {obj_id:7497, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186258}, {obj_id:7498, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179235}, {obj_id:7499, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187845}, {obj_id:7500, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175259}, {obj_id:7501, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188489}, {obj_id:7502, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200494}, {obj_id:7503, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199487}, {obj_id:7504, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207393}, {obj_id:7505, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179176}, {obj_id:7506, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175554}, {obj_id:7507, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194627}, {obj_id:7508, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7509, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7510, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7511, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191972}, {obj_id:7512, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186090}, {obj_id:7513, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7514, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7515, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7516, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7517, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7518, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181649}, {obj_id:7519, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189979}, {obj_id:7520, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185060}, {obj_id:7521, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7522, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7523, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7524, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7525, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234091}, {obj_id:7526, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230643}, {obj_id:7527, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7528, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174080}, {obj_id:7529, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176993}, {obj_id:7530, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183270}, {obj_id:7531, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196560}, {obj_id:7532, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192134}, {obj_id:7533, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189476}, {obj_id:7534, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190737}, {obj_id:7535, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198118}, {obj_id:7536, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194036}, {obj_id:7537, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186988}, {obj_id:7538, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236340}, {obj_id:7539, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234055}, {obj_id:7540, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7541, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7542, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7543, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240841}, {obj_id:7544, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240897}, {obj_id:7545, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7546, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187538}, {obj_id:7547, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195884}, {obj_id:7548, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185189}, {obj_id:7549, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7550, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177480}, {obj_id:7551, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177520}, {obj_id:7552, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7553, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7554, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:7555, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230939}, {obj_id:7556, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210481}, {obj_id:7557, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:267194}, {obj_id:7558, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229175}, {obj_id:7559, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229674}, {obj_id:7560, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262092}, {obj_id:7561, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:227661}]) [2024-04-24 10:00:50.996908] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [874646][T1003_PlanCache][T1003][Y0-0000000000000000-0-0] [lt=403] schedule next cache evict task(evict_interval=5000000) [2024-04-24 10:00:50.997134] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60355}, multi_version_start=1713895332517443000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.997703] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60354}, multi_version_start=1713895332517443000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.998352] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2720) [861318][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4002] get invalid Ethernet speed, use default(devname="eth0") [2024-04-24 10:00:50.998365] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60488}, multi_version_start=1713895332517443000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.998450] WDIAG [SERVER] runTimerTask (ob_server.cpp:3239) [861318][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4002] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4002, ret="OB_INVALID_ARGUMENT") [2024-04-24 10:00:50.998964] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:381}, multi_version_start=1713895332591727001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.999463] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60329}, multi_version_start=1713895332699162001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:50.999664] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [2323545][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=92] current clock is (clock_=0, max_seq_=0, mem_limit=3181127926, cur_mem_hold=297795584, attr_.tenant_id_=1004) [2024-04-24 10:00:51.000016] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101082}, multi_version_start=1713895332699162001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.000537] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:344}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.000746] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [862447][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=91] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:51.001113] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60430}, multi_version_start=1713895332699162001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.001620] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50263}, multi_version_start=1713895332883096000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.002090] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50115}, multi_version_start=1713895332883096001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.002664] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60104}, multi_version_start=1713895332883096001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.003183] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60245}, multi_version_start=1713895332883096001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.003635] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:432}, multi_version_start=1713895333015699000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.004251] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=142] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50239}, multi_version_start=1713895333080464000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.004770] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=98] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1713895333080464000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.005344] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60478}, multi_version_start=1713895333181371002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.005804] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:327}, multi_version_start=1713895333181371002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.006356] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=118] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60248}, multi_version_start=1713895333283595001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.006923] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:410}, multi_version_start=1713895333283595001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.007525] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60147}, multi_version_start=1713895333283595001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.007864] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=78] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=128) [2024-04-24 10:00:51.008043] INFO [LIB] stat (utility.h:1197) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=138] [PALF STAT APPEND COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=0, this=0x14b147074e50, extra_info="id:1") [2024-04-24 10:00:51.008255] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=201] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.008266] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60361}, multi_version_start=1713895333283595003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.008636] INFO [LIB] stat (utility.h:1197) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=110] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=87, this=0x14b1336eb470, extra_info="") [2024-04-24 10:00:51.008780] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=99] [PALF STAT GROUP LOG INFO](palf_id=1001, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=2, avg_log_batch_cnt=1, total_group_log_size=244, avg_group_log_size=122) [2024-04-24 10:00:51.009004] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=80] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, append size=1695) [2024-04-24 10:00:51.009100] INFO [LIB] stat (utility.h:1197) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=78] [PALF STAT APPEND COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=0, this=0x14b134390e50, extra_info="id:1001") [2024-04-24 10:00:51.009301] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=184] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.009347] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=126] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60323}, multi_version_start=1713895333482335001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.009405] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=95] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.009984] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101057}, multi_version_start=1713895333482335001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.010549] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101001}, multi_version_start=1713895333683790000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.011129] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60158}, multi_version_start=1713895333683790000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.011635] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60358}, multi_version_start=1713895333683790000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.012226] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=123] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:423}, multi_version_start=1713895333875950001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.012767] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60334}, multi_version_start=1713895333875950002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.013281] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50261}, multi_version_start=1713895334263327001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.013789] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60296}, multi_version_start=1713895334263327001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.014299] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=166] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60381}, multi_version_start=1713895334418933000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.014950] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60380}, multi_version_start=1713895334418933000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.015473] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=131] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50391}, multi_version_start=1713895334623105000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.016047] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60487}, multi_version_start=1713895334623105000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.017325] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101033}, multi_version_start=1713895334623105000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.017750] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50479}, multi_version_start=1713895334623105000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.018363] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=98] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:316}, multi_version_start=1713895335390590000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.019001] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60251}, multi_version_start=1713895335390590000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.019449] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60293}, multi_version_start=1713895335481827000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.020931] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50392}, multi_version_start=1713895335553346000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.021349] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60181}, multi_version_start=1713895335553346000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.021820] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.022460] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=130] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50305}, multi_version_start=1713895335765358000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.022969] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50121}, multi_version_start=1713895335959450000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.023404] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:232}, multi_version_start=1713895335959450000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.023942] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50401}, multi_version_start=1713895336160193000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.024425] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:499}, multi_version_start=1713895336160193000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.025032] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60119}, multi_version_start=1713895336160193000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.025574] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50004}, multi_version_start=1713895336315793000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.026248] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=111] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50475}, multi_version_start=1713895336315793000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.026751] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=123] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:166}, multi_version_start=1713895336442310002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.027330] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101030}, multi_version_start=1713895336571110000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.027819] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101094}, multi_version_start=1713895336571110000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.028367] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101020}, multi_version_start=1713895336571110000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.028970] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50306}, multi_version_start=1713895336761193000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.029530] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101044}, multi_version_start=1713895336761193000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.030123] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:100002}, multi_version_start=1713895336962058000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.030611] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60452}, multi_version_start=1713895337004240000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.031240] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60386}, multi_version_start=1713895337004240000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.031821] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60133}, multi_version_start=1713895337135545000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.032381] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=128] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50167}, multi_version_start=1713895337135545000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.032913] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=77][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.032906] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50440}, multi_version_start=1713895337364923000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.033576] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=161] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:292}, multi_version_start=1713895337364923000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.034069] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=80] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.034278] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=149] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:293}, multi_version_start=1713895337364923000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.034948] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:290}, multi_version_start=1713895337364923000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.035507] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50146}, multi_version_start=1713895337562995000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.036025] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50207}, multi_version_start=1713895337562995001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.036520] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:371}, multi_version_start=1713895337562995001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.037007] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:388}, multi_version_start=1713895337765690000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.037546] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60450}, multi_version_start=1713895337765690000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.038101] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50455}, multi_version_start=1713895337765690000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.038546] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101013}, multi_version_start=1713895337965758000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.039113] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:192}, multi_version_start=1713895337965758003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.039609] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1713895337965758003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.040164] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60294}, multi_version_start=1713895337965758003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.040840] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60214}, multi_version_start=1713895338166582001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.041411] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=140] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:401}, multi_version_start=1713895338166582001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.042030] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60398}, multi_version_start=1713895338238732001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.042129] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=118] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-04-24 10:00:51.042514] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101072}, multi_version_start=1713895338403656001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.043066] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50458}, multi_version_start=1713895338403656001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.043457] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60432}, multi_version_start=1713895338437469003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.044014] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50431}, multi_version_start=1713895338437469003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.044461] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101088}, multi_version_start=1713895338437469003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.045064] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50150}, multi_version_start=1713895338742431000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.045507] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [862292][T1002_L5_G0][T1002][YB420A000002-000616C2B16C24DE-0-0] [lt=226] is logging [2024-04-24 10:00:51.045589] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101069}, multi_version_start=1713895340458061000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.045644] WDIAG [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:116) [862292][T1002_L5_G0][T1002][YB420A000002-000616C2B16C24DE-0-0] [lt=106][errcode=-4023] submit log callback use too mush time(submit_log_ts=1713924049918819, cb={log_ts:{val:1713924049917601001, v:0}, log_buf:"", pos:40, id_srv_type:0, limited_id:1713924066491232000}, service_type=0, latest_log_ts={val:1713924039886464001, v:0}, self="10.0.0.3:2882") [2024-04-24 10:00:51.045753] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [862292][T1002_L5_G0][T1002][YB420A000002-000616C2B16C24DE-0-0] [lt=103] submit log with lock(ret=-4023, service_type_=0, last_id=1713924066491232000, limited_id=1713924066491232000) [2024-04-24 10:00:51.045858] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [862292][T1002_L5_G0][T1002][YB420A000002-000616C2B16C24DE-0-0] [lt=98] get number(ret=0, service_type_=0, range=1, base_id=1713924051045053000, start_id=1713924051045053000, end_id=1713924051045053001) [2024-04-24 10:00:51.046193] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:244}, multi_version_start=1713895342973186000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.046745] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50357}, multi_version_start=1713895342973186000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.047225] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50245}, multi_version_start=1713895343081229001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.047786] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60493}, multi_version_start=1713895343081229001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.048399] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=166] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50410}, multi_version_start=1713895343263240000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.048763] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [862260][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=78] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1002, min_version={val:1713924048554997000, v:0}, base_version={val:1713924048276279000, v:0}, server_count=2, svr_array=[{addr:"10.0.0.3:2882", version:{val:1713924048276279000, v:0}, valid_part_count:0, total_part_count:3, generate_tstamp:1713924051048632, is_skipped:true}, {addr:"10.0.0.2:2882", version:{val:1713924048554997000, v:0}, valid_part_count:3, total_part_count:3, generate_tstamp:1713924049277905, is_skipped:false}], skip_server_count=1, skip_servers=["10.0.0.3:2882"]) [2024-04-24 10:00:51.048985] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60351}, multi_version_start=1713895343263240001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.049097] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [861771][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=92] current monitor number(seq_=-1) [2024-04-24 10:00:51.049405] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [861573][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=100] LogLoopThread round_cost_time(us)(round_cost_time=283) [2024-04-24 10:00:51.049499] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60106}, multi_version_start=1713895343263240001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.050074] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60402}, multi_version_start=1713895343469278000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.050560] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:191}, multi_version_start=1713895343469278001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.050743] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=98] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.051140] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50249}, multi_version_start=1713895343469278001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.051209] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [862260][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=93] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=4505) [2024-04-24 10:00:51.051316] INFO [LIB] stat (utility.h:1197) [862260][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=78] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x14b14c97ee50, extra_info="id:1") [2024-04-24 10:00:51.051735] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101060}, multi_version_start=1713895343469278001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.052326] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=128] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:252}, multi_version_start=1713895343658170000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.052807] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:279}, multi_version_start=1713895343658170000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.053410] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=148] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:296}, multi_version_start=1713895343658170000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.054000] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60336}, multi_version_start=1713895343806685001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.054444] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:357}, multi_version_start=1713895344005274000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.055013] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50442}, multi_version_start=1713895344120792000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.055544] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101022}, multi_version_start=1713895344167754000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.056057] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:151}, multi_version_start=1713895344167754000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.056594] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50362}, multi_version_start=1713895344167754000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.057186] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=133] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50110}, multi_version_start=1713895344654581001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.057745] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50252}, multi_version_start=1713895344654581001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.058331] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50471}, multi_version_start=1713895344868610000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.058852] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:325}, multi_version_start=1713895344868610001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.059401] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=128] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50226}, multi_version_start=1713895344868610001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.059988] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50488}, multi_version_start=1713895345055997000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.060503] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50214}, multi_version_start=1713895345514436000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.061055] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60412}, multi_version_start=1713895346301396000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.061544] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.062142] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:305}, multi_version_start=1713895346301396000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.062612] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50428}, multi_version_start=1713895346509072000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.063152] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50113}, multi_version_start=1713895346540437001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.063706] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60451}, multi_version_start=1713895346540437001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.064262] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=121] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60404}, multi_version_start=1713895346713988000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.064931] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50441}, multi_version_start=1713895346713988000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.065483] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:389}, multi_version_start=1713895346960363000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.066108] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101035}, multi_version_start=1713895346960363000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.066614] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:5}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.067190] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=86][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.067399] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60327}, multi_version_start=1713895347157056000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.068061] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=122] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50145}, multi_version_start=1713895347428516000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.068604] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60426}, multi_version_start=1713895347428516000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.069300] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101058}, multi_version_start=1713895347428516000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.069942] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=107] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:407}, multi_version_start=1713895347525663001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.070564] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101081}, multi_version_start=1713895347525663001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.071559] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=132] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:245}, multi_version_start=1713895347877859000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.072208] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=118] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101086}, multi_version_start=1713895347877859000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.072441] INFO [STORAGE] exec_wait (ob_tmp_file_cache.cpp:1531) [1556243][T1004_ComTimerT][T1004][Y0-0000000000000000-0-0] [lt=143] succeed to do one round of tmp block io(ret=-4023, loop_nums=0, wait_io_cnt=0, washing_count=0, block_cache_num=0, page_cache_num=0) [2024-04-24 10:00:51.072709] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101052}, multi_version_start=1713895348036219000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.073393] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50133}, multi_version_start=1713895348036219001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.074039] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101019}, multi_version_start=1713895348036219001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.074581] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60157}, multi_version_start=1713895348238153000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.075092] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=133] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:403}, multi_version_start=1713895348238153001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.075688] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101079}, multi_version_start=1713895348238153001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.076306] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50307}, multi_version_start=1713895348386300001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.077005] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50272}, multi_version_start=1713895348657467000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.077605] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:383}, multi_version_start=1713895348657467000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.078215] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=136] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:238}, multi_version_start=1713895348976521001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.078726] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=176] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60306}, multi_version_start=1713895348976521002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.079361] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50333}, multi_version_start=1713895348976521002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.080020] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=107] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:150}, multi_version_start=1713895349153858000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.080205] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=109] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.080328] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=99] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.080495] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=157] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.080710] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=122] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50238}, multi_version_start=1713895349195714000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.081300] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:406}, multi_version_start=1713895349195714000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.081926] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.082515] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60330}, multi_version_start=1713895349909753001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.083153] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50319}, multi_version_start=1713895350393566000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.083592] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50377}, multi_version_start=1713895350393566000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.084256] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:396}, multi_version_start=1713895350393566000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.084931] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60407}, multi_version_start=1713895350525382000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.085468] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50430}, multi_version_start=1713895350525382001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.085640] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [874519][T1004_LCLSender][T1004][Y0-0000000000000000-0-0] [lt=111] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=1, total_destructed_detector=1, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x14b13edd82b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-04-24 10:00:51.086105] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=123] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50104}, multi_version_start=1713895350525382001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.086686] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50425}, multi_version_start=1713895350728536000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.087302] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50480}, multi_version_start=1713895350728536000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.087838] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=107] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60390}, multi_version_start=1713895350818130001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.088354] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50436}, multi_version_start=1713895350818130001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.088961] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50160}, multi_version_start=1713895351121004000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.089284] INFO eloop_run (eloop.c:143) [861399][pnio1][T0][Y0-0000000000000000-0-0] [lt=91] PNIO [ratelimit] time: 1713924051089278, bytes: 14676611010, bw: 0.196900 MB/s, add_ts: 1007557, add_bytes: 208025 [2024-04-24 10:00:51.089432] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:474}, multi_version_start=1713895351134398001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.090039] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60393}, multi_version_start=1713895351387128001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.090537] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50451}, multi_version_start=1713895353019163000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.091190] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50366}, multi_version_start=1713895355993993000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.091769] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60148}, multi_version_start=1713895357110941000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.092360] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=136] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60472}, multi_version_start=1713895357678374000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.093539] INFO [SHARE] deserialize_ (ob_rpc_struct.cpp:9227) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=63] deserialize txDesc from session(tx_desc={this:0x14b17298bf40, tx_id:{txid:176911}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924048513063, op_sn:23, alloc_ts:-1, active_ts:1713924048513603, commit_ts:-1, finish_ts:-1, timeout_us:999996610, lock_timeout_us:-1, expire_ts:1713925048510213, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430290216442775, first_scn:1713924048523055, last_scn:1713924051090113, last_touch_ts:0}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, this={tenant_id:1004, tx_desc:{this:0x14b17298bf40, tx_id:{txid:176911}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924048513063, op_sn:23, alloc_ts:-1, active_ts:1713924048513603, commit_ts:-1, finish_ts:-1, timeout_us:999996610, lock_timeout_us:-1, expire_ts:1713925048510213, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430290216442775, first_scn:1713924048523055, last_scn:1713924051090113, last_touch_ts:0}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, ls_id:{id:1001}, type:3, buf_.length():0x9f, request_id:1, register_flag:{need_flush_redo_instantly_:false, mds_base_scn_:{val:18446744073709551615, v:3}}}) [2024-04-24 10:00:51.093992] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60004}, multi_version_start=1713895358406968000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.094234] INFO [STORAGE] create_tablet (ob_ls_tablet_service.cpp:1972) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=160] no need to create sstable(ls_id={id:1001}, tablet_id={id:1152921504606847582}, create_tablet_schema={ObStorageSchema:{this:0x14b11ebc0060, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:0, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1713924050928856, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[{meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:1, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:1, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}]}, table_id:500928, index_status:1, truncate_version:-1}) [2024-04-24 10:00:51.094387] INFO [STORAGE] init (ob_medium_compaction_mgr.cpp:155) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=141] success to init medium clog recorder(ret=0, ls_id={id:1001}, tablet_id={id:1152921504606847582}, max_saved_version=0) [2024-04-24 10:00:51.094486] INFO [STORAGE.TRANS] init_storage_recorder (ob_tablet_memtable_mgr.cpp:149) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=92] success to init storage recorder(ret=0, ls_id={id:1001}, tablet_id={id:1152921504606847582}, max_saved_schema_version=1713924050928856, max_saved_medium_scn=0, compat_mode=0) [2024-04-24 10:00:51.094539] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:391}, multi_version_start=1713895358845736000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.094715] INFO [STORAGE] init_for_first_time_creation (ob_tablet.cpp:263) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=98] succeeded to init tablet for first time creation(ret=0, sstable=NULL, *this={this:0x14b04a6f1b20, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:2, length:0, tablet_addr:[0-0-0-3458765810900664320-0](offset=0,size=1208,type=3,seq=0), allocator:0x14b0e25e2ad0, tablet_meta:{version:1, ls_id:{id:1001}, tablet_id:{id:1152921504606847582}, data_tablet_id:{id:200301}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1713895200302650000, multi_version_start:1713895200302650000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1713895200302650000, cur_report_version:1713895200302650000, data_checksum:0, row_count:0}, table_store_flag:{with_major_sstable:0}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1713924050928856, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1713924050928856}, table_store_addr:{ptr:{this:0x14b04a6f1fe0, version:257, major_tables:{cnt:0, is_inited:false}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:0}, is_ready_for_read:false, sstable_arrays:{major_tables:{null}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{null}}}, addr:[0-0-0-3458764857417924608-0](offset=0,size=320,type=3,seq=0)}, storage_schema_addr:{ptr:{this:0x14b04a6f2130, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:0, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1713924050928856, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[{meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:1, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:1, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}]}, addr:[0-0-0-3458764814468251648-0](offset=0,size=280,type=3,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:0}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:7, str:"MAX"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:0, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0}) [2024-04-24 10:00:51.095072] INFO [STORAGE.TRANS] create_tablet (ob_tablet_to_ls_cache.cpp:135) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=308] create tablet cache(ret=0, ret="OB_SUCCESS", tablet_id={id:1152921504606847582}, ls_id={id:1001}) [2024-04-24 10:00:51.095163] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [862344][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=137] ====== tenant freeze timer task ====== [2024-04-24 10:00:51.095210] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:426}, multi_version_start=1713895359646480001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.095285] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [862344][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=95] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=96468992, ctx.total_memstore_used_=130023424, ctx.total_memstore_hold_=130023424, memstore_freeze_trigger_limit_=107374180, tenant_id=1001) [2024-04-24 10:00:51.095390] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [862344][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=94] [TenantFreezer] oldest frozen memtable(list="") [2024-04-24 10:00:51.095435] INFO [STORAGE] get_tablet_report_info (ob_tablet.cpp:4405) [861423][TbltTblUp0][T1004][YB420A000003-000616C2AB2901F5-0-0] [lt=91] no major sstables in this tablet, cannot report(ret=-4725) [2024-04-24 10:00:51.095428] INFO [MDS] register_to_mds_table_mgr (mds_table_mgr.cpp:100) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=91] register success(ret=0, ret="OB_SUCCESS", p_mds_table={this:0x14b0d6aad850, ls_id:{id:1001}, tablet_id:{id:1152921504606847582}, flushing_scn:{val:18446744073709551615, v:3}, rec_scn:{val:4611686018427387903, v:0}, last_inner_recycled_scn:{val:0, v:0}, total_node_cnt:0, construct_sequence:12262, debug_info:{do_init_tablet_pointer:0x14b051fd5128, do_remove_tablet_pointer:null, init_ts:"2024-04-24 10:00:51.95379", last_reset_ts:"1970-01-01 08:00:00.0", remove_ts:"1970-01-01 08:00:00.0", last_flush_ts:"1970-01-01 08:00:00.0", switch_to_empty_shell_ts:"1970-01-01 08:00:00.0", init_trace_id:YB420A000002-000616C2C34C0159-0-0, remove_trace_id:Y0-0000000000000000-0-0}}) [2024-04-24 10:00:51.095532] WDIAG [SERVER] inner_fill_tablet_info_ (ob_service.cpp:2451) [861423][TbltTblUp0][T1004][YB420A000003-000616C2AB2901F5-0-0] [lt=82][errcode=-4725] fail to get tablet report info from tablet(ret=-4725, ret="OB_TABLET_NOT_EXIST", tenant_id=1004, tablet_id={id:1152921504606847582}) [2024-04-24 10:00:51.095570] INFO [MDS] init (mds_table_base.cpp:76) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=115] mds table inited(ret=0, ret="OB_SUCCESS", *this={this:0x14b0d6aad850, ls_id:{id:1001}, tablet_id:{id:1152921504606847582}, flushing_scn:{val:18446744073709551615, v:3}, rec_scn:{val:4611686018427387903, v:0}, last_inner_recycled_scn:{val:0, v:0}, total_node_cnt:0, construct_sequence:12262, debug_info:{do_init_tablet_pointer:0x14b051fd5128, do_remove_tablet_pointer:null, init_ts:"2024-04-24 10:00:51.95379", last_reset_ts:"1970-01-01 08:00:00.0", remove_ts:"1970-01-01 08:00:00.0", last_flush_ts:"1970-01-01 08:00:00.0", switch_to_empty_shell_ts:"1970-01-01 08:00:00.0", init_trace_id:YB420A000002-000616C2C34C0159-0-0, remove_trace_id:Y0-0000000000000000-0-0}}) [2024-04-24 10:00:51.095757] WDIAG [SERVER] do_batch_update_ (ob_tablet_table_updater.cpp:613) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F5-0-0] [lt=129][errcode=-4002] tasks num not match or invalid tablet_operator(ret=-4002, ret="OB_INVALID_ARGUMENT", task_cnt=0, replica_cnt=0, checksum_cnt=0) [2024-04-24 10:00:51.095790] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:348}, multi_version_start=1713895360041018000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.095928] INFO [SERVER] do_batch_update_ (ob_tablet_table_updater.cpp:645) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F5-0-0] [lt=97] REPORT: batch update tablets finished(ret=-4002, ret="OB_INVALID_ARGUMENT", replicas.count()=0, cost_time=179) [2024-04-24 10:00:51.096044] WDIAG [SERVER] batch_process_tasks (ob_tablet_table_updater.cpp:514) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F5-0-0] [lt=102][errcode=-4002] do_batch_update_ failed(tmp_ret=-4002, tmp_ret="OB_INVALID_ARGUMENT", start_time=1713924051095390, tasks count=0, tablet replicas count=0) [2024-04-24 10:00:51.096030] INFO [MDS.EVENT]construct_insert_record_user_mds_node_ (mds_row.ipp:216) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=104] WRITE_NODE(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:1152921504606847582}}, event={alloc:null, timestamp:"2024-04-24 10:00:51.95994", event:"WRITE_NODE", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176911, seq_no:0, redo_scn:{val:4611686018427387903, v:0}, end_scn:{val:4611686018427387903, v:0}, trans_version:{val:4611686018427387903, v:0}, node_type:1, state:0}) [2024-04-24 10:00:51.096197] INFO [STORAGE] set_tablet_status (ob_ls_tablet_service.cpp:2382) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=106] succeeded to set tablet status(ret=0, tablet_id={id:1152921504606847582}, tablet_status={tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}) [2024-04-24 10:00:51.096382] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=167] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50385}, multi_version_start=1713895360210033001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.096424] INFO [STORAGE] set_tablet_gc_trigger (ob_tablet_gc_service.cpp:277) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=132] set_tablet_gc_trigger(this={tablet_persist_trigger:2, is_inited:true}) [2024-04-24 10:00:51.096513] INFO [MDS] register_process (ob_tablet_create_mds_helper.cpp:84) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=80] create tablet register(ret=0, ret="OB_SUCCESS", arg={ls_id:{id:1001}, major_frozen_scn:{val:1713895200302650000, v:0}, total_tablet_cnt:1, tablets:{ [874592] [YB420A000002-000616C2C34C0159-0-0] data_tablet_id:{id:200301}, tablet_ids:[{id:1152921504606847582}], compat_mode:0, is_create_bind_hidden_tablets:false }}) [2024-04-24 10:00:51.096671] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=94] notify MDS(ret=0, trans_id_={txid:176911}, ls_id_={id:1001}, notify_type=0, log_ts={val:18446744073709551615, v:3}, notify_array.count()=1, notify_array=[{register_no_:0, has_submitted_:false, has_synced_:false, type:3, data_.length():159}], total_time=0) [2024-04-24 10:00:51.096782] INFO [STORAGE.TRANS] register_mds_into_tx (ob_trans_service.cpp:912) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=80] register multi data source result(ret=0, ret="OB_SUCCESS", arg={tenant_id:1004, tx_desc:{this:0x14b17298bf40, tx_id:{txid:176911}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924048513063, op_sn:23, alloc_ts:-1, active_ts:1713924048513603, commit_ts:-1, finish_ts:-1, timeout_us:999996610, lock_timeout_us:-1, expire_ts:1713925048510213, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430290216442775, first_scn:1713924048523055, last_scn:1713924051090113, last_touch_ts:0}, {id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993242739549473, first_scn:1713924051092566, last_scn:1713924051092566, last_touch_ts:1713924051092568}], exec_info_reap_ts:1713924051092567, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, ls_id:{id:1001}, type:3, buf_.length():0x9f, request_id:1, register_flag:{need_flush_redo_instantly_:false, mds_base_scn_:{val:18446744073709551615, v:3}}}, result={result:0, tx_result:{incomplete:false, parts:[{id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993242739549473, first_scn:1713924051092566, last_scn:1713924051092566, last_touch_ts:1713924051092567}], touched_ls_list:[], cflict_txs:[]}}, tx_desc={this:0x14b17298bf40, tx_id:{txid:176911}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924048513063, op_sn:23, alloc_ts:-1, active_ts:1713924048513603, commit_ts:-1, finish_ts:-1, timeout_us:999996610, lock_timeout_us:-1, expire_ts:1713925048510213, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430290216442775, first_scn:1713924048523055, last_scn:1713924051090113, last_touch_ts:0}, {id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993242739549473, first_scn:1713924051092566, last_scn:1713924051092566, last_touch_ts:1713924051092568}], exec_info_reap_ts:1713924051092567, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, local_retry_cnt=0, retry_cnt=0, request_id=1, time_guard=time guard have no click events for optimization) [2024-04-24 10:00:51.096953] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50438}, multi_version_start=1713895360210033001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.097508] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:235}, multi_version_start=1713895360586155000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.098005] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50336}, multi_version_start=1713895360586155002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.098474] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60401}, multi_version_start=1713895360722056001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.098599] INFO [SHARE] inner_batch_remove_by_sql_ (ob_tablet_replica_checksum_operator.cpp:442) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F5-0-0] [lt=106] will batch delete tablet replica checksum(affected_rows=0) [2024-04-24 10:00:51.099043] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:397}, multi_version_start=1713895360722056001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.099191] INFO [SERVER] do_batch_remove_ (ob_tablet_table_updater.cpp:593) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F5-0-0] [lt=151] REPORT: batch remove tablets finished(ret=0, ret="OB_SUCCESS", tasks_count=1, cost_time=3027) [2024-04-24 10:00:51.099329] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F5-0-0] [lt=113][errcode=-4002] fail to batch process task(ret=-4002) [2024-04-24 10:00:51.099398] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F5-0-0] [lt=61][errcode=-4002] fail to batch execute task(ret=-4002, tasks.count()=1) [2024-04-24 10:00:51.099587] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=98] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:110}, multi_version_start=1713895360993336000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.100174] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60254}, multi_version_start=1713895360993336000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.100692] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=129] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:202}, multi_version_start=1713895361097591000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.101292] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60146}, multi_version_start=1713895361097591000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.101829] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=106] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:319}, multi_version_start=1713895361809321000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.102285] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=241] current clock is (clock_=0, max_seq_=0, mem_limit=3181127926, cur_mem_hold=297795584, attr_.tenant_id_=1004) [2024-04-24 10:00:51.102412] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=125] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50423}, multi_version_start=1713895361809321000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.102943] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:385}, multi_version_start=1713895362179031000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.103450] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60111}, multi_version_start=1713895362179031000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.103947] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50208}, multi_version_start=1713895364193628000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.104475] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50342}, multi_version_start=1713895364193628000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.105016] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50223}, multi_version_start=1713895364420937000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.105531] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60244}, multi_version_start=1713895364420937002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.106012] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60439}, multi_version_start=1713895364661350000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.106461] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50452}, multi_version_start=1713895364661350000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.106932] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60003}, multi_version_start=1713895364661350000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.107141] INFO [PALF] try_reset_last_fetch_log_info_ (log_sliding_window.cpp:1838) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141E-0-0] [lt=89] reset last fetch log info(palf_id=1, self="10.0.0.3:2882", expected_end_lsn={lsn:221691154}) [2024-04-24 10:00:51.107271] INFO [LIB] stat (utility.h:1197) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141E-0-0] [lt=74] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=11, this=0x14b186b47470, extra_info="") [2024-04-24 10:00:51.107410] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141E-0-0] [lt=95] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="FOLLOWER", total_group_log_cnt=4, avg_log_batch_cnt=0, total_group_log_size=3622, avg_group_log_size=905) [2024-04-24 10:00:51.107431] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50349}, multi_version_start=1713895364861720000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.107530] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141E-0-0] [lt=112] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=3216) [2024-04-24 10:00:51.107750] INFO [LIB] stat (utility.h:1197) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141E-0-0] [lt=70] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=37, this=0x14b186b476a0, extra_info="palf_id:1") [2024-04-24 10:00:51.107822] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=122] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.107842] INFO [PALF] read (log_cache.cpp:100) [861714][T1_ReplaySrv6][T1][Y0-0000000000000000-0-0] [lt=85] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3216, hit_cnt=2, read_cnt=2, hit rate=1.000000000000000000e+00) [2024-04-24 10:00:51.108026] INFO [LIB] stat (utility.h:1197) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141E-0-0] [lt=251] [PALF STAT LOG LIFE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=571276, this=0x14b186b478d0, extra_info="palf_id:1") [2024-04-24 10:00:51.108074] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=224] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.108158] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141E-0-0] [lt=126] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=4, avg_log_gen_to_freeze_time=10, avg_log_gen_to_submit_time=103, avg_log_submit_to_flush_time=469783, avg_log_submit_to_slide_time=571173) [2024-04-24 10:00:51.108190] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=108] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.108190] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50446}, multi_version_start=1713895365129129000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.108218] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141E-0-0] [lt=53][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=364245, log_task={header:{begin_lsn:{lsn:179748114}, end_lsn:{lsn:179748652}, log_id:364245, min_scn:{val:1713924048988077000, v:0}, max_scn:{val:1713924048988077000, v:0}, data_len:482, proposal_id:5, prev_lsn:{lsn:179747121}, prev_proposal_id:5, committed_end_lsn:{lsn:179748114}, data_checksum:2484932848, accum_checksum:2618840427, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924048993343, freeze_ts:1713924048993355, submit_ts:1713924048993430, flushed_ts:1713924050718133, gen_to_freeze cost time:12, gen_to_submit cost time:87, submit_to_flush cost time:1724703}, fs_cb_begin_ts=1713924051107643, log_life_time=2114300) [2024-04-24 10:00:51.108719] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:253}, multi_version_start=1713895365129129000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.108956] INFO [LIB] stat (utility.h:1197) [861713][T1_ReplaySrv5][T1][Y0-0000000000000000-0-0] [lt=177] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=420, this=0x14b1976048f8, extra_info="") [2024-04-24 10:00:51.109067] INFO [LIB] stat (utility.h:1197) [861713][T1_ReplaySrv5][T1][Y0-0000000000000000-0-0] [lt=96] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=935, this=0x14b197604b28, extra_info="") [2024-04-24 10:00:51.109290] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:382}, multi_version_start=1713895365288419002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.109587] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C4B-0-0] [lt=114][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=259319, log_task={header:{begin_lsn:{lsn:67159461}, end_lsn:{lsn:67159589}, log_id:259319, min_scn:{val:1713924049917601001, v:0}, max_scn:{val:1713924049917601001, v:0}, data_len:72, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:67159339}, data_checksum:1608530475, accum_checksum:2685460241, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924049918534, freeze_ts:1713924050714205, submit_ts:1713924050714399, flushed_ts:1713924050963852, gen_to_freeze cost time:795671, gen_to_submit cost time:795865, submit_to_flush cost time:249453}, fs_cb_begin_ts=1713924051109528, log_life_time=1190994) [2024-04-24 10:00:51.109754] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:681) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=95] update id meta(ret=0, service_type=0, limited_id=1713924066491232000, latest_scn={val:1713924049917601001, v:0}, *this={tenant_id:1002, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895310268028001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924045940260000, v:0}, tablet_change_checkpoint_scn:{val:1713875507773116000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924066491232000, latest_log_ts:{val:1713924049917601001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713890395635598001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}) [2024-04-24 10:00:51.109827] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60344}, multi_version_start=1713895365288419002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.109926] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:189) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=100] handle submit callback(service_type_=0, success=true, used_ts=1191096, log_ts={val:1713924049917601001, v:0}, limited_id=1713924066491232000) [2024-04-24 10:00:51.109998] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:561) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=66] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2024-04-24 10:00:51.110085] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:599) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=79] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2024-04-24 10:00:51.110207] WDIAG [CLOG] statistics_cb_cost_ (ob_log_apply_service.cpp:933) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=76][errcode=-4389] cb cost too much time(lsn={lsn:67159517}, scn={val:1713924049917601001, v:0}, idx=9, total_cost_time=1191431, append_cost_time=205, cb_wait_thread_time=1191026, cb_wait_commit_time=0, cb_cost_time=200, append_start_time=1713924049918514, append_finish_time=1713924049918719, cb_first_handle_time=1713924051109745, cb_finish_time=1713924051109945) [2024-04-24 10:00:51.110339] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=118] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50157}, multi_version_start=1713895365599180000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.110549] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [862157][T1002_TeRec][T1001][YB420A000003-000616C2AE397391-0-0] [lt=97] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924039746013, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924051109945, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:51.110941] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:350}, multi_version_start=1713895365599180000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.111465] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1713895365599180000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.112041] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60149}, multi_version_start=1713895365807654000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.112197] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD299-0-0] [lt=109][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=304862, log_task={header:{begin_lsn:{lsn:79626650}, end_lsn:{lsn:79626772}, log_id:304862, min_scn:{val:1713924048914830000, v:0}, max_scn:{val:1713924048914830000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:79624410}, data_checksum:667309872, accum_checksum:3770941774, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924049076315, freeze_ts:1713924049076322, submit_ts:1713924049076451, flushed_ts:1713924050714287, gen_to_freeze cost time:7, gen_to_submit cost time:136, submit_to_flush cost time:1637836}, fs_cb_begin_ts=1713924051112148, log_life_time=2035833) [2024-04-24 10:00:51.112383] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2BFC8F2A9-0-0] [lt=204][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924050897126000, v:0}, log_ts:{val:1713924050897126000, v:0}, lsn:{lsn:79626828}, submit_ts:1713924050897126}, this:0x14b1481bc200, is_inited_:true, trans_id:{txid:327202}, ls_id:{id:1}, ctx:0x14b1481b9290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=215240, ctx_lock_wait_time=7) [2024-04-24 10:00:51.112536] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50189}, multi_version_start=1713895365807654000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.113117] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:291}, multi_version_start=1713895366151271000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.113206] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [874729][T1003_LCLSender][T1003][Y0-0000000000000000-0-0] [lt=106] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=1, total_destructed_detector=1, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x14b1703f82b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-04-24 10:00:51.113266] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F7A-0-0] [lt=66][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=482556, log_task={header:{begin_lsn:{lsn:298523302}, end_lsn:{lsn:298524396}, log_id:482556, min_scn:{val:1713924049017564000, v:0}, max_scn:{val:1713924049180202000, v:0}, data_len:1038, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:298522329}, data_checksum:1463097465, accum_checksum:2320389601, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924049018126, freeze_ts:1713924050716624, submit_ts:1713924050716933, flushed_ts:1713924050964239, gen_to_freeze cost time:1698498, gen_to_submit cost time:1698807, submit_to_flush cost time:247306}, fs_cb_begin_ts=1713924051113206, log_life_time=2095080) [2024-04-24 10:00:51.113413] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=93][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924049017564000, v:0}, log_ts:{val:1713924049017564000, v:0}, lsn:{lsn:298523358}, submit_ts:1713924049017564}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:548733}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=2095832, ctx_lock_wait_time=9) [2024-04-24 10:00:51.113659] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50492}, multi_version_start=1713895366442803000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.113712] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=84][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924049180202000, v:0}, log_ts:{val:1713924049180202000, v:0}, lsn:{lsn:298523910}, submit_ts:1713924049180202}, this:0x14b0fc0a5ac0, is_inited_:true, trans_id:{txid:548734}, ls_id:{id:1}, ctx:0x14b0fc0a2b50, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=1933480, ctx_lock_wait_time=21) [2024-04-24 10:00:51.114229] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD29A-0-0] [lt=90][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=330733, log_task={header:{begin_lsn:{lsn:88123610}, end_lsn:{lsn:88123732}, log_id:330733, min_scn:{val:1713924048914830000, v:0}, max_scn:{val:1713924048914830000, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:88121908}, data_checksum:2222604892, accum_checksum:1495036943, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924049076575, freeze_ts:1713924049076587, submit_ts:1713924049076742, flushed_ts:1713924050714591, gen_to_freeze cost time:12, gen_to_submit cost time:167, submit_to_flush cost time:1637849}, fs_cb_begin_ts=1713924051114191, log_life_time=2037616) [2024-04-24 10:00:51.114229] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60471}, multi_version_start=1713895367763401000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.114714] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.115219] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60332}, multi_version_start=1713895367856238001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.115613] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=54] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60405}, multi_version_start=1713895367856238001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.116146] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60150}, multi_version_start=1713895368025520000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.116569] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60213}, multi_version_start=1713895368025520000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.117073] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60375}, multi_version_start=1713895368170554001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.117575] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50188}, multi_version_start=1713895368170554001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.118278] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50403}, multi_version_start=1713895368362163000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.118845] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101043}, multi_version_start=1713895368405387003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.119498] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=237] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.119859] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD29B-0-0] [lt=91][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=299918, log_task={header:{begin_lsn:{lsn:40358733}, end_lsn:{lsn:40360947}, log_id:299918, min_scn:{val:1713924049250797000, v:0}, max_scn:{val:1713924050204218000, v:0}, data_len:2158, proposal_id:5, prev_lsn:{lsn:40358611}, prev_proposal_id:5, committed_end_lsn:{lsn:40358733}, data_checksum:2739249615, accum_checksum:420511841, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924050949302, freeze_ts:1713924050949314, submit_ts:1713924050949455, flushed_ts:1713924050964037, gen_to_freeze cost time:12, gen_to_submit cost time:153, submit_to_flush cost time:14582}, fs_cb_begin_ts=1713924051119798, log_life_time=170496) [2024-04-24 10:00:51.120024] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB902-0-0] [lt=67][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=505465, log_task={header:{begin_lsn:{lsn:314133705}, end_lsn:{lsn:314134313}, log_id:505465, min_scn:{val:1713924048929552000, v:0}, max_scn:{val:1713924048929552001, v:0}, data_len:552, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:314133163}, data_checksum:1500370376, accum_checksum:3952837271, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924048929734, freeze_ts:1713924050715637, submit_ts:1713924050715908, flushed_ts:1713924050954021, gen_to_freeze cost time:1785903, gen_to_submit cost time:1786174, submit_to_flush cost time:238113}, fs_cb_begin_ts=1713924051119961, log_life_time=2190227) [2024-04-24 10:00:51.120149] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:315}, multi_version_start=1713895368405387003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.120241] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C4C-0-0] [lt=131][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=301038, log_task={header:{begin_lsn:{lsn:165104153}, end_lsn:{lsn:165104275}, log_id:301038, min_scn:{val:1713924048889699001, v:0}, max_scn:{val:1713924048889699001, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:165102580}, prev_proposal_id:5, committed_end_lsn:{lsn:165104153}, data_checksum:1715074492, accum_checksum:262579087, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924050955042, freeze_ts:1713924050955048, submit_ts:1713924050955366, flushed_ts:1713924050971124, gen_to_freeze cost time:6, gen_to_submit cost time:324, submit_to_flush cost time:15758}, fs_cb_begin_ts=1713924051120157, log_life_time=165115) [2024-04-24 10:00:51.120399] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=162][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048929552000, v:0}, log_ts:{val:1713924048929552000, v:0}, lsn:{lsn:314133761}, submit_ts:1713924048929552}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:568574}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=2190833, ctx_lock_wait_time=7) [2024-04-24 10:00:51.120679] WDIAG [CLOG] statistics_cb_cost_ (ob_log_apply_service.cpp:933) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=101][errcode=-4389] cb cost too much time(lsn={lsn:314133761}, scn={val:1713924048929552000, v:0}, idx=0, total_cost_time=2190257, append_cost_time=51, cb_wait_thread_time=2190611, cb_wait_commit_time=0, cb_cost_time=-405, append_start_time=1713924048929714, append_finish_time=1713924048929765, cb_first_handle_time=1713924051120376, cb_finish_time=1713924051119971) [2024-04-24 10:00:51.120688] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50378}, multi_version_start=1713895368558770000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.121279] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=120] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50225}, multi_version_start=1713895368558770001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.121935] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=149] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50409}, multi_version_start=1713895368558770001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.122432] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:167}, multi_version_start=1713895368558770001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.122936] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60121}, multi_version_start=1713895368558770001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.123360] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=84] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8899076096, global_cache_size=630472704, tenant_max_wash_size=947814, tenant_min_wash_size=947814, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:51.123476] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:149}, multi_version_start=1713895368765567000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.123991] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60151}, multi_version_start=1713895368765567000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.124555] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50232}, multi_version_start=1713895368765567001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.125130] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50236}, multi_version_start=1713895368765567001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.125661] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:143}, multi_version_start=1713895368967258000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.126280] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:189}, multi_version_start=1713895368967258000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.127034] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=134] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:446}, multi_version_start=1713895368967258000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.127588] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=104] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60246}, multi_version_start=1713895369166257000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.128193] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101032}, multi_version_start=1713895369379679001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.128783] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=118] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60345}, multi_version_start=1713895369379679001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.129365] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=146] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:208}, multi_version_start=1713895369854900000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.129981] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50356}, multi_version_start=1713895369876517000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.130464] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:326}, multi_version_start=1713895369876517000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.131093] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60455}, multi_version_start=1713895369992694000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.131587] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50331}, multi_version_start=1713895370077380000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.132143] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50457}, multi_version_start=1713895370077380000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.132679] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50332}, multi_version_start=1713895370254297000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.133039] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=129][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.133336] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50472}, multi_version_start=1713895370282129000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.133837] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50474}, multi_version_start=1713895370282129000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.134481] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=172] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50248}, multi_version_start=1713895370423988000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.134585] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=91] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.135259] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:107}, multi_version_start=1713895370481552000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.135982] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50324}, multi_version_start=1713895370481552000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.136729] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50432}, multi_version_start=1713895370481552003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.137489] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=253] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60168}, multi_version_start=1713895370693017000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.138211] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=136] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60392}, multi_version_start=1713895370693017002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.138840] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:102}, multi_version_start=1713895370891357002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.139565] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=192] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50296}, multi_version_start=1713895370891357002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.140356] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=133] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60236}, multi_version_start=1713895371024861000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.141081] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:478}, multi_version_start=1713895371060134000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.141529] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=110] notify MDS(ret=0, trans_id_={txid:176911}, ls_id_={id:1}, notify_type=0, log_ts={val:18446744073709551615, v:3}, notify_array.count()=1, notify_array=[{register_no_:0, has_submitted_:false, has_synced_:false, type:1, data_.length():76}], total_time=0) [2024-04-24 10:00:51.141640] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50294}, multi_version_start=1713895371060134000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.142423] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=152] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50358}, multi_version_start=1713895371260682000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.142423] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [874742][T1003_LSArchive][T1003][YB420A000003-000616C2B608F0E7-0-0] [lt=94] gc stale ls task succ [2024-04-24 10:00:51.143040] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50360}, multi_version_start=1713895371260682000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.143654] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101028}, multi_version_start=1713895371260682000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.143673] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [861571][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=97] BlockGCTimerTask success(ret=0, cost_time_us=37, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"10.0.0.3:2882", log_dir:"/data/admin/test/oceanbase/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):171, sequence:2}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:0}}) [2024-04-24 10:00:51.144418] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:376}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.145229] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:239}, multi_version_start=1713895371511376000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.145764] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50253}, multi_version_start=1713895371511376000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.146429] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.147009] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50132}, multi_version_start=1713895372750063000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.147514] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60378}, multi_version_start=1713895372750063000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.148043] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:411}, multi_version_start=1713895372937344000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.148587] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101021}, multi_version_start=1713895372988803000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.149151] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50343}, multi_version_start=1713895372988803000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.149661] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60324}, multi_version_start=1713895373171685000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.150261] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50250}, multi_version_start=1713895373171685000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.150746] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=111] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50487}, multi_version_start=1713895373188600001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.150951] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=118] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.151283] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60292}, multi_version_start=1713895373188600001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.151503] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=95] notify MDS(ret=0, trans_id_={txid:176911}, ls_id_={id:1}, notify_type=0, log_ts={val:18446744073709551615, v:3}, notify_array.count()=1, notify_array=[{register_no_:0, has_submitted_:false, has_synced_:false, type:1, data_.length():76}], total_time=0) [2024-04-24 10:00:51.151913] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60196}, multi_version_start=1713895373188600001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.152537] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=115] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50350}, multi_version_start=1713895373373427000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.153153] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60239}, multi_version_start=1713895373373427000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.153709] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50498}, multi_version_start=1713895373409819001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.154217] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:111}, multi_version_start=1713895373409819001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.154699] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60416}, multi_version_start=1713895373409819001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.155163] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50111}, multi_version_start=1713895373590189001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.155741] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:477}, multi_version_start=1713895373590189001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.155829] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:397) [861321][ServerTracerTim][T0][YB420A000003-000616C2A9491716-0-0] [lt=68] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2024-04-24 10:00:51.156225] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:209}, multi_version_start=1713895373590189001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.156704] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60366}, multi_version_start=1713895373590189003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.157207] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60369}, multi_version_start=1713895373790810000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.157699] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60110}, multi_version_start=1713895374905530000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.158319] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:443}, multi_version_start=1713895374905530000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.158846] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50375}, multi_version_start=1713895375291919000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.159400] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=128] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101037}, multi_version_start=1713895375291919000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.159840] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=72] tx gc loop thread is running(MTL_ID()=1003) [2024-04-24 10:00:51.159925] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:390}, multi_version_start=1713895375291919000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.159977] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=130] try gc retain ctx [2024-04-24 10:00:51.160432] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:263}, multi_version_start=1713895375493236001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.161021] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:135}, multi_version_start=1713895375493236003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.160962] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [874768][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2635-0-0] [lt=99][errcode=-4023] tx is 2pc logging(this={this:0x14b0fbfc0dd0, ref:4, trans_id:{txid:338714}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924080944372, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924050970985], ctx_create_time:1713924050955440}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924050966509, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924050970319000, v:0}, log_ts:{val:1713924050970319000, v:0}, lsn:{lsn:298525467}, submit_ts:1713924050970319}, this:0x14b0fbfc3d40, is_inited_:true, trans_id:{txid:338714}, ls_id:{id:1}, ctx:0x14b0fbfc0dd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338714}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924050970319000, v:0}, start_scn:{val:1713924050970319000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924050970319000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338714} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3469150453 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924050970319000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924050970319, block_frozen_memtable:null}) [2024-04-24 10:00:51.161195] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [874768][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2635-0-0] [lt=219][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fbfc0dd0, ref:4, trans_id:{txid:338714}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924080944372, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924050970985], ctx_create_time:1713924050955440}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924050966509, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924050970319000, v:0}, log_ts:{val:1713924050970319000, v:0}, lsn:{lsn:298525467}, submit_ts:1713924050970319}, this:0x14b0fbfc3d40, is_inited_:true, trans_id:{txid:338714}, ls_id:{id:1}, ctx:0x14b0fbfc0dd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338714}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924050970319000, v:0}, start_scn:{val:1713924050970319000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877380852288 max_table_version=1713877384450672 trans_version={val:1713924050970319000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338714} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3469150453 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=4 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=4 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924050970319000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924050970319, block_frozen_memtable:null}) [2024-04-24 10:00:51.161312] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [874768][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2635-0-0] [lt=108][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338714}) [2024-04-24 10:00:51.161242] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=87] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1003, *ls_ptr={ls_meta:{tenant_id:1003, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895201246660000, v:0}, clog_base_lsn:{lsn:67104768}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1713877340363486001, v:0}, all_id_meta:{id_meta:[{limited_id:1713924059270392000, latest_log_ts:{val:1713924042654251001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713900544098309001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1713877439572285001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:7, palf_env_:0x14b14219e030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-04-24 10:00:51.161387] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [874768][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2635-0-0] [lt=68][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338714}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924051156346, cluster_id:1713874979}, expire_ts:1713924080966169, commit_start_scn:{val:1713924050957910000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:51.161466] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [874768][T1003_L0_G0][T1003][YB420A000002-000616C2B8FC2635-0-0] [lt=65][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338714}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:7, timestamp:1713924051156346, cluster_id:1713874979}, expire_ts:1713924080966169, commit_start_scn:{val:1713924050957910000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:51.161547] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:335}, multi_version_start=1713895375493236003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.162135] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101054}, multi_version_start=1713895375493236003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.162726] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60363}, multi_version_start=1713895375705162000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.162801] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=110] notify MDS(ret=0, trans_id_={txid:176911}, ls_id_={id:1}, notify_type=0, log_ts={val:18446744073709551615, v:3}, notify_array.count()=1, notify_array=[{register_no_:0, has_submitted_:false, has_synced_:false, type:1, data_.length():67}], total_time=0) [2024-04-24 10:00:51.163300] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50444}, multi_version_start=1713895375705162000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.163799] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:153}, multi_version_start=1713895375810006002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.164341] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=129] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50165}, multi_version_start=1713895375810006002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.164829] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60364}, multi_version_start=1713895376106510001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.165404] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=142] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50119}, multi_version_start=1713895376106510001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.166009] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60433}, multi_version_start=1713895376317937000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.166483] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:374}, multi_version_start=1713895376317937001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.166999] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50005}, multi_version_start=1713895376317937001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.167352] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=101][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.167544] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=120] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101041}, multi_version_start=1713895376520729000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.168040] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50369}, multi_version_start=1713895376520729000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.168553] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60370}, multi_version_start=1713895376817717000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.169045] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50399}, multi_version_start=1713895377018378000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.169483] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60210}, multi_version_start=1713895377018378000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.170050] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101024}, multi_version_start=1713895377259689000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.170534] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50147}, multi_version_start=1713895377467723001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.171076] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:236}, multi_version_start=1713895377467723001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.171563] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50316}, multi_version_start=1713895378046324000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.172158] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:237}, multi_version_start=1713895378046324000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.172769] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60225}, multi_version_start=1713895378252416000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.173410] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=141] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60131}, multi_version_start=1713895378252416000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.174069] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50190}, multi_version_start=1713895378412188001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.174641] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60423}, multi_version_start=1713895378461518001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.175265] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50439}, multi_version_start=1713895378461518001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.175832] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60391}, multi_version_start=1713895378659560000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.176425] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=136] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101055}, multi_version_start=1713895378659560000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.177089] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=133] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50400}, multi_version_start=1713895380198550000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.177699] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=98] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60357}, multi_version_start=1713895383652101000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.178413] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:113}, multi_version_start=1713895383894961000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.179104] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60458}, multi_version_start=1713895383894961000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.179683] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50365}, multi_version_start=1713895383894961000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.180396] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=91] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.180405] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=172] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50290}, multi_version_start=1713895384054251001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.180517] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=99] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.180590] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=65] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.181102] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60437}, multi_version_start=1713895384054251001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.181692] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50433}, multi_version_start=1713895384054251001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.182317] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50135}, multi_version_start=1713895384054251001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.182817] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:471}, multi_version_start=1713895384257710000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.183431] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=139] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:450}, multi_version_start=1713895384257710000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.183550] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=201] notify MDS(ret=0, trans_id_={txid:176911}, ls_id_={id:1001}, notify_type=0, log_ts={val:18446744073709551615, v:3}, notify_array.count()=1, notify_array=[{register_no_:0, has_submitted_:false, has_synced_:false, type:1, data_.length():67}], total_time=0) [2024-04-24 10:00:51.184089] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1713895384257710000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.184587] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50376}, multi_version_start=1713895384437004001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.185309] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60397}, multi_version_start=1713895384803565000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.185820] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:375}, multi_version_start=1713895384803565001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.186462] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=177] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:3}, multi_version_start=1713895384803565001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.187141] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60360}, multi_version_start=1713895385003503000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.187729] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60112}, multi_version_start=1713895385208684000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.188319] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60333}, multi_version_start=1713895385757651000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.188841] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:168}, multi_version_start=1713895385757651000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.189470] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=158] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:475}, multi_version_start=1713895385757651000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.190144] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60399}, multi_version_start=1713895385966644000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.190621] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101040}, multi_version_start=1713895386118071000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.191246] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:259}, multi_version_start=1713895386118071000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.191830] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=98] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50491}, multi_version_start=1713895386461153000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.192566] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=179] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:248}, multi_version_start=1713895386461153001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.193223] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=126] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60153}, multi_version_start=1713895386461153001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.193805] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=119] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50386}, multi_version_start=1713895386772931002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.194444] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=136] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101047}, multi_version_start=1713895386994133000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.195179] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=106] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50416}, multi_version_start=1713895386994133000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.195845] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=149] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50265}, multi_version_start=1713895387715499000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.196526] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=138] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:106}, multi_version_start=1713895388433990000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.196987] INFO [STORAGE.TRANS] print_statistics (ob_xa_define.cpp:467) [861682][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=111] xa statistics(total_active_xa_ctx_count=0, total_standby_clearup_count=0, total_success_xa_start=0, total_failure_xa_start=0, total_success_xa_prepare=0, total_failure_xa_prepare=0, total_success_xa_1pc_commit=0, total_failure_xa_1pc_commit=0, total_success_xa_2pc_commit=0, total_failure_xa_2pc_commit=0, total_failure_xa_rollback=0, total_success_dblink_promotion=0, total_failure_dblink_promotion=0, total_success_dblink=0, total_failure_dblink=0) [2024-04-24 10:00:51.197151] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60307}, multi_version_start=1713895388433990000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.197345] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:445) [874708][T1003_SSTableDe][T1003][Y0-0000000000000000-0-0] [lt=90] skip defragment task(ret=0, need_defragment=false) [2024-04-24 10:00:51.197486] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:485) [874708][T1003_SSTableDe][T1003][Y0-0000000000000000-0-0] [lt=133] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=3) [2024-04-24 10:00:51.197581] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:503) [874708][T1003_SSTableDe][T1003][Y0-0000000000000000-0-0] [lt=85] successfully defragment data blocks(rewrite_cnt=0) [2024-04-24 10:00:51.197766] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50411}, multi_version_start=1713895388621094000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.198331] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50459}, multi_version_start=1713895388621094000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.198967] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=133] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50389}, multi_version_start=1713895388761984000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.199537] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=106] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101059}, multi_version_start=1713895388761984000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.200691] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50131}, multi_version_start=1713895388961443000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.201190] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50262}, multi_version_start=1713895389870833001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.201739] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:324}, multi_version_start=1713895389870833001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.202360] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:441}, multi_version_start=1713895389870833001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.203066] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=122] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:130}, multi_version_start=1713895390093477000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.203633] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:262}, multi_version_start=1713895390093477000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.204232] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=130] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50143}, multi_version_start=1713895390302276000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.204806] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:114}, multi_version_start=1713895390302276000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.205442] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=154] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50374}, multi_version_start=1713895390457128000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.206099] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60240}, multi_version_start=1713895390494515001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.206587] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:206}, multi_version_start=1713895390494515001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.207279] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=156] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:207}, multi_version_start=1713895390697796001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.207765] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:498}, multi_version_start=1713895390697796001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.208050] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=64] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.208187] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=94] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.208297] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=96] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.208375] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:380}, multi_version_start=1713895391061499000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.208987] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.209625] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=148] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60413}, multi_version_start=1713895391061499000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.210281] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50345}, multi_version_start=1713895391261976000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.210989] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50166}, multi_version_start=1713895391261976000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.211499] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60356}, multi_version_start=1713895391261976000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.212170] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=104] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50434}, multi_version_start=1713895391261976003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.212800] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=122] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:295}, multi_version_start=1713895391475546001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.213389] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=124] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60371}, multi_version_start=1713895391696629000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.213963] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50180}, multi_version_start=1713895391696629000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.214545] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60343}, multi_version_start=1713895391696629000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.215237] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=122] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60143}, multi_version_start=1713895391875773000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.215767] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=109] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60406}, multi_version_start=1713895391875773000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.216308] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:370}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.216581] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=192] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1003, min_version={val:1713924048824341000, v:0}, base_version={val:1713924048824341000, v:0}, server_count=2, svr_array=[{addr:"10.0.0.2:2882", version:{val:1713924048824341000, v:0}, valid_part_count:0, total_part_count:1, generate_tstamp:1713924051120616, is_skipped:true}, {addr:"10.0.0.3:2882", version:{val:1713924048824341000, v:0}, valid_part_count:0, total_part_count:1, generate_tstamp:1713924051216463, is_skipped:true}], skip_server_count=2, skip_servers=["10.0.0.2:2882", "10.0.0.3:2882"]) [2024-04-24 10:00:51.216787] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=170] no need to update min/max version(new_version={val:1713924048824341000, v:0}, min_version={val:1713924048824341000, v:0}, max_version={val:1713924048824341000, v:0}) [2024-04-24 10:00:51.216989] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=111] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60255}, multi_version_start=1713895392084003000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.217602] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50242}, multi_version_start=1713895392084003000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.217846] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [874526][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=95] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1004, min_version={val:1713924048598120000, v:0}, base_version={val:1713924048463546001, v:0}, server_count=2, svr_array=[{addr:"10.0.0.2:2882", version:{val:1713924048598120000, v:0}, valid_part_count:3, total_part_count:3, generate_tstamp:1713924049277446, is_skipped:false}, {addr:"10.0.0.3:2882", version:{val:1713924048510522000, v:0}, valid_part_count:0, total_part_count:3, generate_tstamp:1713924051217713, is_skipped:true}], skip_server_count=1, skip_servers=["10.0.0.3:2882"]) [2024-04-24 10:00:51.218183] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=133] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:436}, multi_version_start=1713895392562685000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.218768] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101046}, multi_version_start=1713895392649603000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.219383] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=145] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60427}, multi_version_start=1713895392901858000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.219762] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [874526][T1004_TenantWea][T1003][YB420A000003-000616C2B8CCACF1-0-0] [lt=122] current clock is (clock_=0, max_seq_=0, mem_limit=264912214, cur_mem_hold=121634816, attr_.tenant_id_=1003) [2024-04-24 10:00:51.220088] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861495][RpcIO][T0][Y0-0000000000000000-0-0] [lt=92] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:51.220096] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=127] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60265}, multi_version_start=1713895393771740000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.220679] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:430}, multi_version_start=1713895394090102000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.221002] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861496][RpcIO][T0][Y0-0000000000000000-0-0] [lt=80] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:51.221031] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861497][RpcIO][T0][Y0-0000000000000000-0-0] [lt=87] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:51.221385] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60105}, multi_version_start=1713895394090102000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.221473] INFO [SERVER] switch_schema (ob_service.cpp:1313) [861736][T1_L0_G0][T1][YB420A000002-000616C2AD7C034D-0-0] [lt=64] start to switch schema(arg=schema_info:{schema_version:1713924046573584, tenant_id:1004, sequence_id:5497558139158}, force_refresh:true, is_async:false) [2024-04-24 10:00:51.221732] INFO [SERVER] async_refresh_schema (ob_server_schema_updater.cpp:484) [861736][T1_L0_G0][T1][YB420A000002-000616C2AD7C034D-0-0] [lt=133] schedule async refresh schema task(ret=0, ret="OB_SUCCESS", tenant_id=1004, schema_version=1713924046573584) [2024-04-24 10:00:51.221837] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [874754][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=132] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:51.222008] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60252}, multi_version_start=1713895394090102000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.221965] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2465) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=104] [REFRESH_SCHEMA] start to refresh and add schema(tenant_ids=[1004]) [2024-04-24 10:00:51.222138] INFO [SHARE.SCHEMA] refresh_tenant_schema (ob_multi_version_schema_service.cpp:2795) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=144] [REFRESH_SCHEMA] start to refresh and add schema by tenant(tenant_id=1004) [2024-04-24 10:00:51.222563] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861498][RpcIO][T0][Y0-0000000000000000-0-0] [lt=70] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:51.222601] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:272}, multi_version_start=1713895394278557000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.222641] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=114] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1001, min_version={val:1713924048441114000, v:0}, base_version={val:1713924048441114000, v:0}, server_count=2, svr_array=[{addr:"10.0.0.3:2882", version:{val:1713924048641803000, v:0}, valid_part_count:0, total_part_count:1, generate_tstamp:1713924051222510, is_skipped:true}, {addr:"10.0.0.2:2882", version:{val:1713924048641803001, v:0}, valid_part_count:0, total_part_count:1, generate_tstamp:1713924051119953, is_skipped:true}], skip_server_count=2, skip_servers=["10.0.0.3:2882", "10.0.0.2:2882"]) [2024-04-24 10:00:51.222745] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=97] no need to update min/max version(new_version={val:1713924048441114000, v:0}, min_version={val:1713924048441114000, v:0}, max_version={val:1713924048441114000, v:0}) [2024-04-24 10:00:51.223223] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50295}, multi_version_start=1713895394278557000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.223840] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50382}, multi_version_start=1713895394445969000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.224269] INFO [SHARE.SCHEMA] refresh_schema (ob_server_schema_service.cpp:5426) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=102] [REFRESH_SCHEMA] start to refresh increment schema(current schema_version=1713924044295272, schema_status={tenant_id:1004, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-24 10:00:51.224490] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=140] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50394}, multi_version_start=1713895394485907001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.225142] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50152}, multi_version_start=1713895394485907001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.225754] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60475}, multi_version_start=1713895394685010000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.226307] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:251}, multi_version_start=1713895394728352000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.226857] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50490}, multi_version_start=1713895394887127000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.227388] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=144] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60191}, multi_version_start=1713895394887127000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.227992] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:223}, multi_version_start=1713895395130043000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.228513] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50291}, multi_version_start=1713895395636692000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.229135] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101045}, multi_version_start=1713895395636692000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.229638] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50371}, multi_version_start=1713895396004353000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.230127] INFO [SQL.JO] compute_table_location (ob_join_order.cpp:278) [2323545][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=91] succeed to calculate base table sharding info(table_id=18446744073709551614, ref_table_id=343, is_global_index=false) [2024-04-24 10:00:51.230183] INFO [SHARE.SCHEMA] check_core_schema_change_ (ob_server_schema_service.cpp:6566) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=85] core schema is not changed(ret=0, ret="OB_SUCCESS", schema_status={tenant_id:1004, snapshot_timestamp:-1, readable_schema_version:-1}, core_schema_version=1713877451559520, new_core_schema_version=1713877451559520) [2024-04-24 10:00:51.230257] INFO [SQL.JO] init_est_sel_info_for_access_path (ob_join_order.cpp:12189) [2323545][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=70] total rowcount, use statistics(table_meta_info_.table_row_count_=654, table_meta_info_.average_row_size_=1.200000000000000000e+02, table_meta_info_.micro_block_count_=0, table_meta_info_.part_size_=7.848000000000000000e+04) [2024-04-24 10:00:51.230283] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:488}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.230799] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60331}, multi_version_start=1713895396159436000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.231409] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=161] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:131}, multi_version_start=1713895396159436000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.232100] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60442}, multi_version_start=1713895396529136001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.232604] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60284}, multi_version_start=1713895396529136001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.233183] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=89] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1004, epoch:-1, self:"10.0.0.3:2882", switch_to_leader_ts:-1}) [2024-04-24 10:00:51.233249] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=134] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50243}, multi_version_start=1713895396800469000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.233297] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=99] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1004, epoch:-1, self:"10.0.0.3:2882", switch_to_leader_ts:-1}) [2024-04-24 10:00:51.233382] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=72][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.233980] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60316}, multi_version_start=1713895396823529002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.234591] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=148] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60182}, multi_version_start=1713895396823529002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.234917] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=192] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.235290] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=120] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:246}, multi_version_start=1713895397153517001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.235805] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60499}, multi_version_start=1713895397153517001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.236425] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=135] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101078}, multi_version_start=1713895397153517001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.237038] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=104] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:148}, multi_version_start=1713895397352090001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.237577] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:408}, multi_version_start=1713895397352090001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.238118] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101049}, multi_version_start=1713895397352090001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.238632] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=132] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:247}, multi_version_start=1713895397561126000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.239238] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=128] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:365}, multi_version_start=1713895397581564000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.239731] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:220}, multi_version_start=1713895397581564000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.240315] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50151}, multi_version_start=1713895397762319000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.240851] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50209}, multi_version_start=1713895398037629000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.241396] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=123] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:221}, multi_version_start=1713895398037629000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.241988] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:366}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.242470] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50255}, multi_version_start=1713895398037629000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.242520] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=87] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-04-24 10:00:51.243135] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60135}, multi_version_start=1713895398237285000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.243746] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=138] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:169}, multi_version_start=1713895398237285000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.244422] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=115] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50404}, multi_version_start=1713895398385070002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.244996] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50202}, multi_version_start=1713895398385070002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.245444] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:213}, multi_version_start=1713895398385070002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.246030] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50330}, multi_version_start=1713895398607086000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.246490] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:119}, multi_version_start=1713895398607086000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.247054] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50412}, multi_version_start=1713895398607086000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.247686] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101091}, multi_version_start=1713895398786349000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.248343] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50426}, multi_version_start=1713895398786349000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.248744] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD29C-0-0] [lt=184][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=304864, log_task={header:{begin_lsn:{lsn:79627270}, end_lsn:{lsn:79627427}, log_id:304864, min_scn:{val:1713924050897126001, v:0}, max_scn:{val:1713924050897126001, v:0}, data_len:101, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:79626650}, data_checksum:4263971082, accum_checksum:615201025, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924050932918, freeze_ts:1713924050964348, submit_ts:1713924050964510, flushed_ts:1713924050976367, gen_to_freeze cost time:31430, gen_to_submit cost time:31592, submit_to_flush cost time:11857}, fs_cb_begin_ts=1713924051248668, log_life_time=315750) [2024-04-24 10:00:51.248818] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60220}, multi_version_start=1713895398786349000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.249018] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=179][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048839739001, v:0}, log_ts:{val:1713924050897126001, v:0}, lsn:{lsn:79627326}, submit_ts:1713924050931982}, this:0x14b1481c2ab0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x400, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=317004, ctx_lock_wait_time=23) [2024-04-24 10:00:51.249152] INFO [STORAGE.TRANS] del_tx_ctx (ob_trans_ctx_mgr_v4.cpp:1314) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=103] can not del a retain ctx(ctx={this:0x14b1481c0010, ref:2, trans_id:{txid:176888}, tenant_id:1004, is_exiting:true, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924048664959], ctx_create_time:1713924045155303}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:50, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79589464}, {lsn:79618713}], redo_log_no:2, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:6, data_.length():1313}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048664122000, v:0}, trans_type:2, next_log_entry_no:5, max_applied_log_ts:{val:1713924050897126001, v:0}, max_applying_log_ts:{val:1713924050897126001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:1713924048659397, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:79627326}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:18}, is_leaf():false, is_root():true, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b1481c2f80, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1}, ctx:0x14b1481c0010, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:2, state:"COMMIT", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048272353001, v:0}, end_scn:{val:1713924048839739001, v:0}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:true}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877443458344 max_table_version=1713877467338312 trans_version={val:1713924048664122000, v:0} commit_version={val:1713924048664122000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=-6233 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1 callback_alloc_count=134 callback_free_count=134 checksum=141527899 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=158 redo_sync_succ_count=158 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=158, slave=0, merge=0, tx_end=158, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:70, retain_cause:0, 2pc_role:0, collected:[], rec_log_ts:{val:1713924048272353001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048665345, block_frozen_memtable:null}) [2024-04-24 10:00:51.249409] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=185] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:356}, multi_version_start=1713895398975592000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.250015] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101039}, multi_version_start=1713895399023080000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.250584] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=139][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67159589}], scn_array=[{val:1713924050922254000, v:0}], curr_size=122, accum_size=250, time_cost=242563) [2024-04-24 10:00:51.250779] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=129] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101029}, multi_version_start=1713895399023080000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.251132] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=111] tx gc loop thread is running(MTL_ID()=1) [2024-04-24 10:00:51.251236] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=95] try gc retain ctx [2024-04-24 10:00:51.251245] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=93][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=259320, log_task={header:{begin_lsn:{lsn:67159589}, end_lsn:{lsn:67159711}, log_id:259320, min_scn:{val:1713924050922254000, v:0}, max_scn:{val:1713924050922254000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:67159461}, data_checksum:667309872, accum_checksum:2194832310, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924051007597, freeze_ts:1713924051007605, submit_ts:1713924051007804, flushed_ts:1713924051250963, gen_to_freeze cost time:8, gen_to_submit cost time:207, submit_to_flush cost time:243159}, fs_cb_begin_ts=1713924051251171, log_life_time=243574) [2024-04-24 10:00:51.251348] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60385}, multi_version_start=1713895399023080000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.252052] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=139] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:283}, multi_version_start=1713895399023080000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.252083] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=118][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:40360947}], scn_array=[{val:1713924050784468000, v:0}], curr_size=122, accum_size=2336, time_cost=130832) [2024-04-24 10:00:51.252276] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=68] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895204295552002, v:0}, clog_base_lsn:{lsn:67104768}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1713875366171467000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924058636985000, latest_log_ts:{val:1713924041985593001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713918035834957001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1713903554859671002, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:2, proposal_id:0, palf_env_:0x14b1945f8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-04-24 10:00:51.252502] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=194] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.252821] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=175] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60408}, multi_version_start=1713895399185117000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.253431] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=178] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:354}, multi_version_start=1713895399185117000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.253983] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60235}, multi_version_start=1713895399243143000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.254496] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=61] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60215}, multi_version_start=1713895399243143000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.255112] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60325}, multi_version_start=1713895399243143000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.255584] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50397}, multi_version_start=1713895399388995000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.256209] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:433}, multi_version_start=1713895399446735000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.256771] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=109] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60374}, multi_version_start=1713895399446735000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.257351] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60261}, multi_version_start=1713895399596429000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.257846] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60206}, multi_version_start=1713895399596429002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.258396] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=128] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:333}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.258940] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50308}, multi_version_start=1713895399596429005, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.259471] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50206}, multi_version_start=1713895399821196001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.260112] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=127] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50413}, multi_version_start=1713895399955572000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.260662] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:402}, multi_version_start=1713895399955572000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.261180] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:424}, multi_version_start=1713895399998116000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.261459] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F7B-0-0] [lt=91][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=482557, log_task={header:{begin_lsn:{lsn:298524396}, end_lsn:{lsn:298525411}, log_id:482557, min_scn:{val:1713924050955440000, v:0}, max_scn:{val:1713924050955440000, v:0}, data_len:959, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:298523302}, data_checksum:8987066, accum_checksum:922545024, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924050956522, freeze_ts:1713924050964276, submit_ts:1713924050964464, flushed_ts:1713924050976539, gen_to_freeze cost time:7754, gen_to_submit cost time:7942, submit_to_flush cost time:12075}, fs_cb_begin_ts=1713924051261395, log_life_time=304873) [2024-04-24 10:00:51.261724] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:100004}, multi_version_start=1713895399998116000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.262217] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=99][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924050955440000, v:0}, log_ts:{val:1713924050955440000, v:0}, lsn:{lsn:298524452}, submit_ts:1713924050955440}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:548735}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=306757, ctx_lock_wait_time=12) [2024-04-24 10:00:51.262384] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60250}, multi_version_start=1713895400199512000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.262607] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=85][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924050970319000, v:0}, log_ts:{val:1713924050970319000, v:0}, lsn:{lsn:298525467}, submit_ts:1713924050970319}, this:0x14b0fbfc3d40, is_inited_:true, trans_id:{txid:338714}, ls_id:{id:1}, ctx:0x14b0fbfc0dd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=292265, ctx_lock_wait_time=12) [2024-04-24 10:00:51.262788] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [874525][T1004_ReqMemEvi][T1004][Y0-0000000000000000-0-0] [lt=109] sql audit evict task end(request_manager_->get_tenant_id()=1004, evict_high_mem_level=278313880, evict_high_size_level=90000, evict_batch_count=0, elapse_time=11, size_used=80643, mem_used=131023872) [2024-04-24 10:00:51.263049] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=133] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60389}, multi_version_start=1713895400199512001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.263214] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD29E-0-0] [lt=109][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=330734, log_task={header:{begin_lsn:{lsn:88123732}, end_lsn:{lsn:88123889}, log_id:330734, min_scn:{val:1713924048914830001, v:0}, max_scn:{val:1713924048914830001, v:0}, data_len:101, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:88123610}, data_checksum:2741952230, accum_checksum:2522090140, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924050933682, freeze_ts:1713924050933692, submit_ts:1713924050933960, flushed_ts:1713924050964701, gen_to_freeze cost time:10, gen_to_submit cost time:278, submit_to_flush cost time:30741}, fs_cb_begin_ts=1713924051263159, log_life_time=329477) [2024-04-24 10:00:51.263577] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874554][T1004_ApplySrv2][T1004][YB420A000003-000616C2BFC8F285-0-0] [lt=139][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924048839739001, v:0}, log_ts:{val:1713924048914830001, v:0}, lsn:{lsn:88123788}, submit_ts:1713924050933369}, this:0x14b055e208a8, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x400, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=330189, ctx_lock_wait_time=8) [2024-04-24 10:00:51.263664] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60477}, multi_version_start=1713895400199512001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.263783] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD29D-0-0] [lt=163][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=299919, log_task={header:{begin_lsn:{lsn:40360947}, end_lsn:{lsn:40361069}, log_id:299919, min_scn:{val:1713924050784468000, v:0}, max_scn:{val:1713924050784468000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:40358733}, prev_proposal_id:5, committed_end_lsn:{lsn:40360947}, data_checksum:1715074492, accum_checksum:1379001982, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924051119641, freeze_ts:1713924051119680, submit_ts:1713924051119752, flushed_ts:1713924051252420, gen_to_freeze cost time:39, gen_to_submit cost time:111, submit_to_flush cost time:132668}, fs_cb_begin_ts=1713924051263703, log_life_time=144062) [2024-04-24 10:00:51.263744] INFO [STORAGE.TRANS] del_tx_ctx (ob_trans_ctx_mgr_v4.cpp:1314) [874554][T1004_ApplySrv2][T1004][YB420A000003-000616C2BFC8F285-0-0] [lt=101] can not del a retain ctx(ctx={this:0x14b055e1e050, ref:2, trans_id:{txid:176888}, tenant_id:1004, is_exiting:true, trans_expired_time:1713925045138248, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924048589222}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:50, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88120890}], redo_log_no:1, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():572}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924048658150001, v:0}, trans_type:2, next_log_entry_no:4, max_applied_log_ts:{val:1713924048914830001, v:0}, max_applying_log_ts:{val:1713924048914830001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:88123788}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88121631}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:18}, is_leaf():true, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e20fc0, is_inited_:true, trans_id:{txid:176888}, ls_id:{id:1001}, ctx:0x14b055e1e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176888}, ref_cnt:2, state:"COMMIT", commit_version:{val:1713924048664122000, v:0}, start_scn:{val:1713924048588043001, v:0}, end_scn:{val:1713924048823139001, v:0}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:true}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=0 trans_version={val:1713924048658150001, v:0} commit_version={val:1713924048664122000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=-6233 tx_status=0 is_readonly=false ref=0 trans_id={txid:176888} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=1 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88121631}}, {id_:{id:1}, offset_:{lsn:79620195}}], upstream_state:70, retain_cause:0, 2pc_role:2, collected:[], rec_log_ts:{val:1713924048588043001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924048589222, block_frozen_memtable:null}) [2024-04-24 10:00:51.264546] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101070}, multi_version_start=1713895400357746000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.264630] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2AE-0-0] [lt=119][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924050938294000, v:0}, log_ts:{val:1713924050938294000, v:0}, lsn:{lsn:88123945}, submit_ts:1713924050938294}, this:0x14b148367dc0, is_inited_:true, trans_id:{txid:327203}, ls_id:{id:1001}, ctx:0x14b148364e50, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=326306, ctx_lock_wait_time=11) [2024-04-24 10:00:51.265289] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=160] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.266114] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:435}, multi_version_start=1713895400399319001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.266835] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=115] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:456}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.267527] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=74][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.267519] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=181] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60249}, multi_version_start=1713895400502968000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.268207] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:254}, multi_version_start=1713895400601126002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.268925] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=115] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101075}, multi_version_start=1713895400601126002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.269589] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50182}, multi_version_start=1713895400601126002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.269600] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [862437][T1001_LSArchive][T1001][YB420A000003-000616C2AEF8F0E7-0-0] [lt=104] gc stale ls task succ [2024-04-24 10:00:51.270428] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=130] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60373}, multi_version_start=1713895400720000003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.271199] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60197}, multi_version_start=1713895400720000003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.271838] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:181}, multi_version_start=1713895400720000003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.272346] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [2322437][T1004_L0_G0][T1004][YB420A000003-000616C2C128F119-0-0] [lt=122] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924035815932, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924051272311, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:51.272529] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=158] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101077}, multi_version_start=1713895401005660000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.273239] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.273179] INFO [SHARE] deserialize_ (ob_rpc_struct.cpp:9227) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=117] deserialize txDesc from session(tx_desc={this:0x14b188558b20, tx_id:{txid:176911}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924048513063, op_sn:36, alloc_ts:-1, active_ts:1713924048513603, commit_ts:-1, finish_ts:-1, timeout_us:999996610, lock_timeout_us:-1, expire_ts:1713925048510213, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430290216442775, first_scn:1713924048523055, last_scn:1713924051162101, last_touch_ts:0}, {id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993242739549473, first_scn:1713924051092566, last_scn:1713924051182812, last_touch_ts:0}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, this={tenant_id:1004, tx_desc:{this:0x14b188558b20, tx_id:{txid:176911}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924048513063, op_sn:36, alloc_ts:-1, active_ts:1713924048513603, commit_ts:-1, finish_ts:-1, timeout_us:999996610, lock_timeout_us:-1, expire_ts:1713925048510213, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430290216442775, first_scn:1713924048523055, last_scn:1713924051162101, last_touch_ts:0}, {id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993242739549473, first_scn:1713924051092566, last_scn:1713924051182812, last_touch_ts:0}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, ls_id:{id:1}, type:6, buf_.length():0x44b, request_id:1, register_flag:{need_flush_redo_instantly_:false, mds_base_scn_:{val:18446744073709551615, v:3}}}) [2024-04-24 10:00:51.273461] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=175] notify MDS(ret=0, trans_id_={txid:176911}, ls_id_={id:1}, notify_type=0, log_ts={val:18446744073709551615, v:3}, notify_array.count()=1, notify_array=[{register_no_:0, has_submitted_:false, has_synced_:false, type:6, data_.length():1099}], total_time=0) [2024-04-24 10:00:51.273576] INFO [STORAGE.TRANS] register_mds_into_tx (ob_trans_service.cpp:912) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=102] register multi data source result(ret=0, ret="OB_SUCCESS", arg={tenant_id:1004, tx_desc:{this:0x14b188558b20, tx_id:{txid:176911}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924048513063, op_sn:36, alloc_ts:-1, active_ts:1713924048513603, commit_ts:-1, finish_ts:-1, timeout_us:999996610, lock_timeout_us:-1, expire_ts:1713925048510213, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430290216442775, first_scn:1713924048523055, last_scn:1713924051272316, last_touch_ts:1713924051272318}, {id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993242739549473, first_scn:1713924051092566, last_scn:1713924051182812, last_touch_ts:0}], exec_info_reap_ts:1713924051272317, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, ls_id:{id:1}, type:6, buf_.length():0x44b, request_id:1, register_flag:{need_flush_redo_instantly_:false, mds_base_scn_:{val:18446744073709551615, v:3}}}, result={result:0, tx_result:{incomplete:false, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430290216442775, first_scn:1713924048523055, last_scn:1713924051272316, last_touch_ts:1713924051272317}], touched_ls_list:[], cflict_txs:[]}}, tx_desc={this:0x14b188558b20, tx_id:{txid:176911}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924048513063, op_sn:36, alloc_ts:-1, active_ts:1713924048513603, commit_ts:-1, finish_ts:-1, timeout_us:999996610, lock_timeout_us:-1, expire_ts:1713925048510213, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430290216442775, first_scn:1713924048523055, last_scn:1713924051272316, last_touch_ts:1713924051272318}, {id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993242739549473, first_scn:1713924051092566, last_scn:1713924051182812, last_touch_ts:0}], exec_info_reap_ts:1713924051272317, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, local_retry_cnt=0, retry_cnt=0, request_id=1, time_guard=time guard have no click events for optimization) [2024-04-24 10:00:51.273956] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:142}, multi_version_start=1713895401005660000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.274483] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50387}, multi_version_start=1713895401204068000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.275280] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=111] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101071}, multi_version_start=1713895401254517000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.275918] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:190}, multi_version_start=1713895401254517000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.276436] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101093}, multi_version_start=1713895401254517000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.276950] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101012}, multi_version_start=1713895401403663000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.277400] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50220}, multi_version_start=1713895401403663001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.278089] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:360}, multi_version_start=1713895401403663001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.278628] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:364}, multi_version_start=1713895401403663001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.279262] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101065}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.279802] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:317}, multi_version_start=1713895401603383000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.280396] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=123] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:438}, multi_version_start=1713895401603383000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.280780] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=86] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.280937] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:214}, multi_version_start=1713895401603383003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.280937] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=141] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.281003] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=59] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.281536] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:394}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.282101] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:329}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.282579] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.283154] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=131] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60372}, multi_version_start=1713895401805083003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.283636] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:100003}, multi_version_start=1713895402005375001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.284272] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=142] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60435}, multi_version_start=1713895402005375001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.284838] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=119] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:105}, multi_version_start=1713895402005375001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.285515] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=190] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.285580] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=56] notify MDS(ret=0, trans_id_={txid:176911}, ls_id_={id:1}, notify_type=2, log_ts={val:18446744073709551615, v:3}, notify_array.count()=4, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:false, type:1, data_.length():76}, {register_no_:2, has_submitted_:true, has_synced_:false, type:1, data_.length():76}, {register_no_:3, has_submitted_:true, has_synced_:false, type:1, data_.length():67}, {register_no_:4, has_submitted_:true, has_synced_:false, type:6, data_.length():1099}], total_time=0) [2024-04-24 10:00:51.286067] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1713895402210136001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.286580] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50435}, multi_version_start=1713895402210136001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.286635] INFO [STORAGE.TRANS] submit_multi_data_source_ (ob_trans_part_ctx.cpp:6456) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=123] submit MDS redo with barrier or base_scn successfully(ret=0, trans_id_={txid:176911}, ls_id_={id:1001}, log_cb={ObTxBaseLogCb:{base_ts:{val:0, v:0}, log_ts:{val:1713924051274688001, v:0}, lsn:{lsn:88128808}, submit_ts:1713924051285620}, this:0x14b055e308a8, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1001}, ctx:0x14b055e2e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():2, range_array_:[{register_no_:1, has_submitted_:true, has_synced_:false, type:3, data_.length():159}, {register_no_:2, has_submitted_:true, has_synced_:false, type:1, data_.length():67}]}, cb_arg_array_:[{log_type:0x4, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, mds_cache_={unsubmitted_size_:0, mds_list_.size():2, max_register_no_:2}, exec_info_.multi_data_source_=[], mds_base_scn={val:0, v:0}, barrier_type=2) [2024-04-24 10:00:51.286823] INFO [MDS.EVENT]before_prepare_ (mds_node.ipp:174) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=114] BEFORE_PREPARE(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:1152921504606847582}}, event={alloc:null, timestamp:"2024-04-24 10:00:51.286801", event:"BEFORE_PREPARE", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176911, seq_no:0, redo_scn:{val:4611686018427387903, v:0}, end_scn:{val:4611686018427387903, v:0}, trans_version:{val:0, v:0}, node_type:1, state:1}) [2024-04-24 10:00:51.287003] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=159] notify MDS(ret=0, trans_id_={txid:176911}, ls_id_={id:1001}, notify_type=2, log_ts={val:18446744073709551615, v:3}, notify_array.count()=2, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:false, type:3, data_.length():159}, {register_no_:2, has_submitted_:true, has_synced_:false, type:1, data_.length():67}], total_time=0) [2024-04-24 10:00:51.287147] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=106] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101014}, multi_version_start=1713895402381548000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.287726] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=106] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60308}, multi_version_start=1713895402436436000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.288281] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1713895402436436000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.288827] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50443}, multi_version_start=1713895402554464000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.289374] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=140] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:261}, multi_version_start=1713895402554464000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.289941] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:147}, multi_version_start=1713895402554464000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.290411] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50103}, multi_version_start=1713895402812328000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.291220] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:358}, multi_version_start=1713895402862974000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.291726] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101004}, multi_version_start=1713895402862974000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.292244] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60456}, multi_version_start=1713895402862974000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.292731] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60473}, multi_version_start=1713895403012554000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.293193] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:457}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.293752] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60160}, multi_version_start=1713895403077203000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.294268] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60394}, multi_version_start=1713895403077203000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.294728] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:120}, multi_version_start=1713895403208015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.295266] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60319}, multi_version_start=1713895403208015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.295733] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.296350] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=239] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60326}, multi_version_start=1713895403208015001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.296968] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:379}, multi_version_start=1713895403397732000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.297536] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=120] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50326}, multi_version_start=1713895403397732000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.298147] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50221}, multi_version_start=1713895403413542001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.298609] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60283}, multi_version_start=1713895403413542001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.299281] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50396}, multi_version_start=1713895403413542001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.299793] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:413}, multi_version_start=1713895403612387001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.300360] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=145] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50235}, multi_version_start=1713895403877247000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.300797] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:240}, multi_version_start=1713895404111058000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.301298] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50213}, multi_version_start=1713895405178572002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.301716] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:133}, multi_version_start=1713895405178572003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.302239] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101050}, multi_version_start=1713895405178572003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.302631] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60362}, multi_version_start=1713895405178572003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.303126] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50407}, multi_version_start=1713895405462445000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.303631] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=117] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:409}, multi_version_start=1713895405668995001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.304165] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:100001}, multi_version_start=1713895405668995001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.304626] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50284}, multi_version_start=1713895405668995001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.305122] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:243}, multi_version_start=1713895405868465000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.305580] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60498}, multi_version_start=1713895405868465001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.306132] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50327}, multi_version_start=1713895405868465001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.306721] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60436}, multi_version_start=1713895405999908000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.307410] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50359}, multi_version_start=1713895405999908000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.307951] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60223}, multi_version_start=1713895406022563002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.308161] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=78] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.308260] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=81] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.308346] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=77] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.308477] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101056}, multi_version_start=1713895406022563002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.308992] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=54] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50210}, multi_version_start=1713895406269808000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.309505] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:386}, multi_version_start=1713895406307073000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.310038] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50383}, multi_version_start=1713895406307073000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.310600] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:434}, multi_version_start=1713895406307073000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.311124] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60400}, multi_version_start=1713895406402493001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.311601] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50244}, multi_version_start=1713895406529549002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.312194] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:180}, multi_version_start=1713895406529549002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.312703] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:490}, multi_version_start=1713895406529549002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.313198] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=75] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:399}, multi_version_start=1713895406529549002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.313682] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60207}, multi_version_start=1713895406674557000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.314215] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=115] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101023}, multi_version_start=1713895406674557000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.314699] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50317}, multi_version_start=1713895406674557000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.315221] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:146}, multi_version_start=1713895406674557000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.315641] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101007}, multi_version_start=1713895406873436000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.316160] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=89] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101048}, multi_version_start=1713895406873436000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.316743] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=171] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.317274] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101026}, multi_version_start=1713895407071442000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.317983] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50372}, multi_version_start=1713895407077510002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.318584] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=82] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50260}, multi_version_start=1713895407077510002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.319195] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50402}, multi_version_start=1713895407276796000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.319609] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60189}, multi_version_start=1713895407883580000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.320122] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50211}, multi_version_start=1713895408081613002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.320548] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50354}, multi_version_start=1713895408081613002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.321105] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50215}, multi_version_start=1713895408306716000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.321550] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50348}, multi_version_start=1713895408641331001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.322078] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60335}, multi_version_start=1713895408641331001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.322552] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60501}, multi_version_start=1713895408641331001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.323133] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=126] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:428}, multi_version_start=1713895408828841000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.323576] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50456}, multi_version_start=1713895408828841001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.324244] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=53] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50106}, multi_version_start=1713895408828841001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.324497] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=128] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8896978944, global_cache_size=630472704, tenant_max_wash_size=947814, tenant_min_wash_size=947814, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:51.324717] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101025}, multi_version_start=1713895409028491000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.325270] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=76] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60188}, multi_version_start=1713895409028491001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.325715] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=98] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50196}, multi_version_start=1713895409028491001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.326208] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=69] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60132}, multi_version_start=1713895409028491001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.326676] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:104}, multi_version_start=1713895409229079000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.327207] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.327709] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=109] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:4}, multi_version_start=1713895409229079001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.328234] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:480}, multi_version_start=1713895409430098000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.328705] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:100005}, multi_version_start=1713895409488267000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.329175] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:355}, multi_version_start=1713895409488267000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.329627] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60224}, multi_version_start=1713895409630702000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.330333] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=136] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:351}, multi_version_start=1713895409630702000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.330957] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60120}, multi_version_start=1713895409808533000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.331518] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60377}, multi_version_start=1713895409826431002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.332106] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:160}, multi_version_start=1713895409826431002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.332690] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50398}, multi_version_start=1713895409826431002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.333171] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60115}, multi_version_start=1713895410042620000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.333421] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=55][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.333729] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50259}, multi_version_start=1713895410042620001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.334286] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60242}, multi_version_start=1713895410243532002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.334822] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:476}, multi_version_start=1713895411039758000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.334982] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=69] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.335406] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=137] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50254}, multi_version_start=1713895411291956000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.335994] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=99] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50370}, multi_version_start=1713895411291956000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.336449] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:260}, multi_version_start=1713895411488818000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.337052] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101085}, multi_version_start=1713895411679117000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.337496] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:387}, multi_version_start=1713895411679117000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.337313] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2648) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=144] [PALF_DUMP](palf_id=1, self="10.0.0.3:2882", [SlidingWindow]={palf_id:1, self:"10.0.0.3:2882", lsn_allocator:{max_log_id:259321, max_lsn:67160209, max_scn:{val:1713924051045053000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:67159711}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:25217169}}, last_submit_lsn:{lsn:67159711}, last_submit_end_lsn:{lsn:67160209}, last_submit_log_id:259321, last_submit_log_pid:5, max_flushed_lsn:{lsn:67159589}, max_flushed_end_lsn:{lsn:67159711}, max_flushed_log_pid:5, committed_end_lsn:{lsn:67159711}, last_slide_log_id:259320, last_slide_scn:{val:1713924050922254000, v:0}, last_slide_lsn:{lsn:67159589}, last_slide_end_lsn:{lsn:67159711}, last_slide_log_pid:5, last_slide_log_accum_checksum:2194832310, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x14b1470aa530}, [StateMgr]={this:0x14b1470ae2b0, self:"10.0.0.3:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:5}, leader:"10.0.0.3:2882", leader_epoch:8, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:64863204}, scan_disk_log_finished:true, last_check_start_id:259321, is_changing_config_with_arb:false, reconfirm_start_time_us:1713921655678945, palf_role_change_cb:0x14b1470b4b70, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"10.0.0.3:2882", alive_paxos_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, alive_paxos_replica_num:2, log_ms_meta:{version:3, proposal_id:5, prev:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:5, config_seq:6}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:4, prev_lsn:{lsn:64863082}, prev_mode_pid:5}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:4, prev_lsn:{lsn:64863082}, prev_end_lsn:{lsn:64863204}, prev_mode_pid:5}, reconfig_barrier:{prev_log_proposal_id:4, prev_lsn:{lsn:64863082}, prev_end_lsn:{lsn:64863204}, prev_mode_pid:5}, persistent_config_version:{proposal_id:5, config_seq:6}, ms_ack_list:0, resend_config_version:{proposal_id:5, config_seq:6}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x14b1470abc70}, [ModeMgr]={palf_id:1, self:"10.0.0.3:2882", applied_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1713921656180363, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:5}, log_config_meta:{version:3, proposal_id:5, prev:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:5, config_seq:6}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:4, prev_lsn:{lsn:64863082}, prev_mode_pid:5}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:73728}, readable_log_tail:{lsn:73728}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/data/admin/test/oceanbase/store/clog/tenant_1002/1/meta", dir_fd:153, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67031040, block_header_serialize_buf_:0x14b1470b0438, flashback_version:0}, log_storage:{log_tail:{lsn:67159711}, readable_log_tail:{lsn:67159711}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1713924008805678001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/data/admin/test/oceanbase/store/clog/tenant_1002/1/log", dir_fd:159, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:67049825, block_header_serialize_buf_:0x14b1470b20b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1713921655679771, this:0x14b1470ae9f0}, [Reconfirm]={palf_id:1, self:"10.0.0.3:2882", state:"FINISHED", new_proposal_id:5, prepare_log_ack_list:1"10.0.0.2:2882";, curr_paxos_follower_list:1{server:"10.0.0.2:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"10.0.0.3:2882", majority_max_accept_pid:4, majority_max_lsn:{lsn:64863204}, saved_end_lsn:{lsn:64863204}, last_submit_prepare_req_time_us:1713921655681059, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:1713921656279620, last_purge_throttling_time_us:-1, this:0x14b1470ae430}) [2024-04-24 10:00:51.337635] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2652) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=314] [PALF_DUMP](palf_id=1, self="10.0.0.3:2882", ack_info_list=[{member:{server:"10.0.0.2:2882", timestamp:1, flag:0}, last_ack_time_us:1713924051248314, last_flushed_end_lsn:{lsn:67159711}}, {member:{server:"10.0.0.3:2882", timestamp:1, flag:0}, last_ack_time_us:1713924051250980, last_flushed_end_lsn:{lsn:67159711}}]) [2024-04-24 10:00:51.337788] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4290) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=144] current log_task status(palf_id=1, self="10.0.0.3:2882", log_id=259321, log_task={header:{begin_lsn:{lsn:67159711}, end_lsn:{lsn:67160209}, log_id:259321, min_scn:{val:1713924051045053000, v:0}, max_scn:{val:1713924051045053000, v:0}, data_len:442, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:67159711}, data_checksum:2423324012, accum_checksum:1204586448, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1713924051046004, freeze_ts:1713924051250991, submit_ts:1713924051251143, flushed_ts:-1, gen_to_freeze cost time:204987, gen_to_submit cost time:205139, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"10.0.0.2:2882", timestamp:1, flag:0}, last_ack_time_us:1713924051248314, last_flushed_end_lsn:{lsn:67159711}}, {member:{server:"10.0.0.3:2882", timestamp:1, flag:0}, last_ack_time_us:1713924051250980, last_flushed_end_lsn:{lsn:67159711}}]) [2024-04-24 10:00:51.338037] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [874706][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=100] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:769, free_obj_hold_cnt:74, allocator used:3351768, allocator total:3466624}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:72, free_obj_hold_cnt:32, allocator used:6821464, allocator total:6821464}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():154752, full allocator used:1600, full allocator total:220288}, tablets_mem=10508376, tablets_mem_limit=257698020, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:23, free_obj_hold_cnt:163, allocator used:369024, allocator total:392448}, pending_cnt=0, wait_gc_count=0, tablet count=841) [2024-04-24 10:00:51.338140] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=124] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60209}, multi_version_start=1713895411679117000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.337964] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2648) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=135] [PALF_DUMP](palf_id=1002, self="10.0.0.3:2882", [SlidingWindow]={palf_id:1002, self:"10.0.0.3:2882", lsn_allocator:{max_log_id:301040, max_lsn:165105970, max_scn:{val:1713924050962740001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:165104275}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:123162930}}, last_submit_lsn:{lsn:165105848}, last_submit_end_lsn:{lsn:165105970}, last_submit_log_id:301040, last_submit_log_pid:5, max_flushed_lsn:{lsn:165104153}, max_flushed_end_lsn:{lsn:165104275}, max_flushed_log_pid:5, committed_end_lsn:{lsn:165104275}, last_slide_log_id:301038, last_slide_scn:{val:1713924048889699001, v:0}, last_slide_lsn:{lsn:165104153}, last_slide_end_lsn:{lsn:165104275}, last_slide_log_pid:5, last_slide_log_accum_checksum:262579087, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x14b13adc2530}, [StateMgr]={this:0x14b13adc62b0, self:"10.0.0.3:2882", palf_id:1002, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:5}, leader:"10.0.0.2:2882", leader_epoch:8, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:301039, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x14b13adccb70, allow_vote:true}, [ConfigMgr]={palf_id:1002, self:"10.0.0.3:2882", alive_paxos_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, alive_paxos_replica_num:2, log_ms_meta:{version:3, proposal_id:5, prev:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:5, config_seq:6}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:4, prev_lsn:{lsn:156707309}, prev_mode_pid:5}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:5, config_seq:6}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1713924048022649, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1713924048022649, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x14b13adc3c70}, [ModeMgr]={palf_id:1002, self:"10.0.0.3:2882", applied_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1713875572331196000, v:0}}, accepted_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1713875572331196000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1713875572331196000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1002, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1713875572331196001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:5}, log_config_meta:{version:3, proposal_id:5, prev:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:5, config_seq:6}, log_sync_memberlist:2{server:"10.0.0.2:2882", timestamp:1, flag:0}{server:"10.0.0.3:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:4, prev_lsn:{lsn:156707309}, prev_mode_pid:5}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1713875572331196000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:5, mode_version:0, access_mode:1, ref_scn:{val:1713875572331196000, v:0}}}, log_meta_storage:{log_tail:{lsn:69632}, readable_log_tail:{lsn:69632}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:201314304}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/data/admin/test/oceanbase/store/clog/tenant_1002/1002/meta", dir_fd:165, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67035136, block_header_serialize_buf_:0x14b13adc8438, flashback_version:0}, log_storage:{log_tail:{lsn:165104275}, readable_log_tail:{lsn:165104275}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:1713915029911414001, v:0}, curr_block_id:2, palf_id:1002}, block_mgr:{log_dir:"/data/admin/test/oceanbase/store/clog/tenant_1002/1002/log", dir_fd:166, min_block_id:0, max_block_id:3, curr_writable_block_id:2}, logical_block_size_:67104768, curr_block_writable_size_:36210029, block_header_serialize_buf_:0x14b13adca0b8, flashback_version:0}, palf_epoch:3, last_purge_throttling_ts:1713921656706735, this:0x14b13adc69f0}, [Reconfirm]={palf_id:1002, self:"10.0.0.3:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x14b13adc6430}) [2024-04-24 10:00:51.338247] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2742) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=274][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="10.0.0.3:2882", tmp_server={server:"10.0.0.2:2882", timestamp:1, flag:0}) [2024-04-24 10:00:51.338357] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2652) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=99] [PALF_DUMP](palf_id=1002, self="10.0.0.3:2882", ack_info_list=[]) [2024-04-24 10:00:51.338471] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2742) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=102][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="10.0.0.3:2882", tmp_server={server:"10.0.0.2:2882", timestamp:1, flag:0}) [2024-04-24 10:00:51.338556] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4290) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=73] current log_task status(palf_id=1002, self="10.0.0.3:2882", log_id=301039, log_task={header:{begin_lsn:{lsn:165104275}, end_lsn:{lsn:165105848}, log_id:301039, min_scn:{val:1713924050962740000, v:0}, max_scn:{val:1713924050962740000, v:0}, data_len:1517, proposal_id:5, prev_lsn:{lsn:165104153}, prev_proposal_id:5, committed_end_lsn:{lsn:165104275}, data_checksum:1891944955, accum_checksum:4112143148, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713924051120015, freeze_ts:1713924051120024, submit_ts:1713924051120123, flushed_ts:-1, gen_to_freeze cost time:9, gen_to_submit cost time:108, submit_to_flush cost time:0}, ack_info_list=[]) [2024-04-24 10:00:51.338737] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50335}, multi_version_start=1713895412042176000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.339311] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:361}, multi_version_start=1713895412042176000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.339858] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:241}, multi_version_start=1713895412042176000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.340396] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=140] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60388}, multi_version_start=1713895412702728000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.341062] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=81] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:306}, multi_version_start=1713895412702728000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.341619] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60443}, multi_version_start=1713895412702728000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.342252] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=134] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:132}, multi_version_start=1713895412856176001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.342784] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101087}, multi_version_start=1713895412856176002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.343292] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50153}, multi_version_start=1713895412856176002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.343813] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60403}, multi_version_start=1713895412856176002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.344313] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=138] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101003}, multi_version_start=1713895412856176002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.344847] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60490}, multi_version_start=1713895413062460000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.345506] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=157] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:336}, multi_version_start=1713895413062460000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.346118] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50233}, multi_version_start=1713895413062460002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.346610] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50427}, multi_version_start=1713895413062460002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.347142] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60315}, multi_version_start=1713895413062460002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.347610] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=92] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50191}, multi_version_start=1713895413257949001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.348277] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:343}, multi_version_start=1713895413273242000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.348838] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50501}, multi_version_start=1713895413273242000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.349441] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=165] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60232}, multi_version_start=1713895413273242000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.349943] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=63] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101051}, multi_version_start=1713895413273242000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.350023] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [874735][T1003_ReqMemEvi][T1003][Y0-0000000000000000-0-0] [lt=73] sql audit evict task end(request_manager_->get_tenant_id()=1003, evict_high_mem_level=19327352, evict_high_size_level=90000, evict_batch_count=0, elapse_time=17, size_used=8181, mem_used=14558208) [2024-04-24 10:00:51.350491] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:100006}, multi_version_start=1713895413457416001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.351060] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50473}, multi_version_start=1713895413457416001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.351349] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=101] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.351560] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60480}, multi_version_start=1713895413457416001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.352262] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=138] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50158}, multi_version_start=1713895413457416004, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.352827] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60382}, multi_version_start=1713895413704831000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.353514] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=161] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50477}, multi_version_start=1713895414734084000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.354239] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=96] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60396}, multi_version_start=1713895415940770000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.354973] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=120] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60237}, multi_version_start=1713895416835739001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.355529] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101061}, multi_version_start=1713895416835739001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.356103] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60202}, multi_version_start=1713895417404347000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.356705] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=158] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50181}, multi_version_start=1713895417436325000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.357331] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60211}, multi_version_start=1713895417604429000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.357846] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50241}, multi_version_start=1713895417604429000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.358444] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=143] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60479}, multi_version_start=1713895417604429000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.359010] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101076}, multi_version_start=1713895417836276000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.359393] INFO [CLOG] operator() (ob_log_replay_service.cpp:1590) [861661][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=94] get_replay_process success(id={id:1}, replayed_log_size=179748652, unreplayed_log_size=0) [2024-04-24 10:00:51.359513] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50168}, multi_version_start=1713895417836276001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.359502] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:159) [861661][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=72] dump tenant replay process(tenant_id=1, unreplayed_log_size(MB)=0, estimate_time(second)=0, replayed_log_size(MB)=171, last_replayed_log_size(MB)=171, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2024-04-24 10:00:51.360092] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=68] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:416}, multi_version_start=1713895417836276001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.360839] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=125] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.361779] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=171] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50107}, multi_version_start=1713895418021780001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.362537] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=170] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50363}, multi_version_start=1713895418021780001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.363136] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101006}, multi_version_start=1713895418215968000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.363737] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60272}, multi_version_start=1713895418215968000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.364244] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:250}, multi_version_start=1713895418215968000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.364799] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.365404] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=163] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50222}, multi_version_start=1713895418418095002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.366052] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60446}, multi_version_start=1713895418466941001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.366612] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60114}, multi_version_start=1713895418466941001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.367270] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=128] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50283}, multi_version_start=1713895418617489000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.367748] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=83][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.367827] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:425}, multi_version_start=1713895418617489000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.367913] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:954) [861320][SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=125] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=669876224, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=403865600, wash_ratio=6, tenant_id=1001) [2024-04-24 10:00:51.368110] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:323) [861320][SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=104] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2024-04-24 10:00:51.368216] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1202) [861320][SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=92] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=162, drift_size_=0, pre_drift_size=0, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2024-04-24 10:00:51.368338] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:954) [861320][SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=81] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=967315456, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=106426368, wash_ratio=6, tenant_id=1002) [2024-04-24 10:00:51.368354] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=130] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50388}, multi_version_start=1713895418617489002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.368579] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:323) [861320][SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=138] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2024-04-24 10:00:51.368671] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1202) [861320][SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=82] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=926, drift_size_=0, pre_drift_size=0, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2024-04-24 10:00:51.368850] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:954) [861320][SqlMemTimer][T1003][Y0-0000000000000000-0-0] [lt=135] trace max work area(auto_calc=true, tenant_max_memory_limit=1288490188, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=64424509, max_wa_memory_size=64424509, tmp_max_wa_memory_size=64424509, pre_mem_target=64424509, remain_memory_size=64424509, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=671387648, washable_size=-2, max_workarea_memory_size=64424509, max_tenant_memory_size=617102540, wash_ratio=6, tenant_id=1003) [2024-04-24 10:00:51.369052] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60221}, multi_version_start=1713895418617489002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.369063] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:323) [861320][SqlMemTimer][T1003][Y0-0000000000000000-0-0] [lt=136] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=8053063, calc_global_bound_size=1048576000000, mem_target_=64424509, wa_max_memory_size=64424509, profile_cnt_=0, total_memory_size=0, max_wa_size=64424509, avg_bound_size=8053063, max_bound_size=8053063, min_bound_size_=1048576) [2024-04-24 10:00:51.369157] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1202) [861320][SqlMemTimer][T1003][Y0-0000000000000000-0-0] [lt=86] timer to calc global bound size(ret=0, global_bound_size_=8053063, manual_calc_cnt_=109, drift_size_=0, pre_drift_size=0, wa_max_memory_size=64424509, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1003, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=8053063, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=64424509, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2024-04-24 10:00:51.369278] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:954) [861320][SqlMemTimer][T1004][Y0-0000000000000000-0-0] [lt=91] trace max work area(auto_calc=true, tenant_max_memory_limit=11596411700, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=579820585, max_wa_memory_size=579820585, tmp_max_wa_memory_size=579820585, pre_mem_target=579820585, remain_memory_size=579820585, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1638285312, washable_size=-2, max_workarea_memory_size=579820585, max_tenant_memory_size=9958126388, wash_ratio=6, tenant_id=1004) [2024-04-24 10:00:51.369436] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:323) [861320][SqlMemTimer][T1004][Y0-0000000000000000-0-0] [lt=88] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=72477573, calc_global_bound_size=1048576000000, mem_target_=579820585, wa_max_memory_size=579820585, profile_cnt_=0, total_memory_size=0, max_wa_size=579820585, avg_bound_size=72477573, max_bound_size=72477573, min_bound_size_=1048576) [2024-04-24 10:00:51.369539] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1202) [861320][SqlMemTimer][T1004][Y0-0000000000000000-0-0] [lt=90] timer to calc global bound size(ret=0, global_bound_size_=72477573, manual_calc_cnt_=640, drift_size_=0, pre_drift_size=0, wa_max_memory_size=579820585, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1004, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=72477573, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=579820585, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2024-04-24 10:00:51.369560] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=74] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50247}, multi_version_start=1713895418821083000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.370142] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101074}, multi_version_start=1713895418886913000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.370698] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60192}, multi_version_start=1713895418886913000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.371290] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60376}, multi_version_start=1713895419019949000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.371830] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50323}, multi_version_start=1713895419019949000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.372468] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=177] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:479}, multi_version_start=1713895419019949000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.373123] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101002}, multi_version_start=1713895419019949000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.373462] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=124] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: MICRO_BLOCK_CACHE_IO, mode: read, size: 4096.00, iops: 11.93, rt: 628.17) [2024-04-24 10:00:51.373597] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=72] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101053}, multi_version_start=1713895419214621000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.373583] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=91] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:51.374177] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:211}, multi_version_start=1713895419214621000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.374256] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=165] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 915.00) [2024-04-24 10:00:51.374344] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=62] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:51.374643] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=79] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50493}, multi_version_start=1713895419229596002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.375148] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:834) [862160][T1002_BalanceEx][T1002][YB420A000003-000616C2B3D9ADA2-0-0] [lt=94] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2024-04-24 10:00:51.375293] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:171) [862160][T1002_BalanceEx][T1002][YB420A000003-000616C2B3D9ADA2-0-0] [lt=136] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2024-04-24 10:00:51.375472] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=123] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50105}, multi_version_start=1713895419229596002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.375715] INFO [COMMON] print_io_status (ob_io_manager.cpp:1393) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=72] [IO STATUS SYS](tenant_id=1002, io_status=sys_group_name: SLOG_IO, mode: write, size: 4096.00, iops: 0.99, rt: 1189954.00) [2024-04-24 10:00:51.375816] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=76] [IO STATUS CONFIG](tenant_id=1002, ref_cnt=2, io_config={group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:1}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:51.376049] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50237}, multi_version_start=1713895419422480000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.376528] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60142}, multi_version_start=1713895419422480000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.377050] INFO [COMMON] print_io_status (ob_io_manager.cpp:1393) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=119] [IO STATUS SYS](tenant_id=1004, io_status=sys_group_name: SLOG_IO, mode: write, size: 4096.00, iops: 0.99, rt: 1189067.00) [2024-04-24 10:00:51.377175] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=103] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60409}, multi_version_start=1713895419422480000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.377189] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=104] [IO STATUS CONFIG](tenant_id=1004, ref_cnt=3, io_config={group_num_:0, memory_limit_:11596411700, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:2}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:51.377330] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=120] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-24 10:00:51.377496] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=135] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-24 10:00:51.377823] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50130}, multi_version_start=1713895419622568000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.378590] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=166] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60322}, multi_version_start=1713895419622568000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.379323] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=146] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:398}, multi_version_start=1713895419655441001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.379954] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:359}, multi_version_start=1713895419655441001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.380528] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50352}, multi_version_start=1713895419752686001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.380833] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=159] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.381043] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=169] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.381148] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=91] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.381279] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60107}, multi_version_start=1713895419845591002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.381990] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50390}, multi_version_start=1713895419845591002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.382058] INFO [STORAGE.TRANS] handle_tx_commit_timeout (ob_trans_service_v4.cpp:393) [861656][T1_TransTimeWhe][T1][Y0-0000000000000000-0-0] [lt=103] handle tx commit timeout(ret=0, tx_id={txid:169795}, ref_cnt=4, cb_executed=false) [2024-04-24 10:00:51.382629] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=105] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50112}, multi_version_start=1713895420021104001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.383392] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=137] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:491}, multi_version_start=1713895420021104001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.384165] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=138] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60474}, multi_version_start=1713895420156133000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.384983] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50322}, multi_version_start=1713895420224546001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.385127] INFO [STORAGE.TRANS] handle_tx_commit_result_ (ob_trans_service_v4.cpp:549) [861401][pnio1][T1][YB420A000003-000616C2ACD902D6-0-0] [lt=70] handle tx commit result(ret=0, ref_cnt_0=4, tx={this:0x14b186574fe0, tx_id:{txid:169795}, state:5, addr:"10.0.0.3:2882", tenant_id:1, session_id:3221540160, assoc_session_id:3221540160, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924049402283, op_sn:4, alloc_ts:1713924049403545, active_ts:1713924049403545, commit_ts:1713924049409760, finish_ts:-1, timeout_us:86400000000, lock_timeout_us:-1, expire_ts:1714010449403545, coord_id:{id:1}, parts:[{id:{id:1}, addr:"10.0.0.2:2882", epoch:1430291097262351, first_scn:1713924049406085, last_scn:1713924049406085, last_touch_ts:1}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:9, commit_cb:0x14b1187d4988, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:1713924059410543, commit_task_.is_registered():true, ref:4}, commit_fin=false, result=-4023) [2024-04-24 10:00:51.385299] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:426) [862224][T1002_FreInfoRe][T1002][Y0-0000000000000000-0-0] [lt=131] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2024-04-24 10:00:51.385299] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2366) [861401][pnio1][T1][YB420A000003-000616C2ACD902D6-0-0] [lt=157][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:169795}, sender_ls_id={id:9223372036854775807}, receiver_ls_id={id:1}, msg_type=20, status=-4023, receiver_addr="10.0.0.2:2882", request_id=4) [2024-04-24 10:00:51.385425] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:285) [861401][pnio1][T0][YB420A000003-000616C2ACD902D6-0-0] [lt=119][errcode=0] trx rpc callback(ret=0, status=-4023, dst="10.0.0.2:2882", result={status:-4023, send_timestamp:1713924051381731, private_data:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:51.385555] INFO [STORAGE] update_info (ob_tenant_freeze_info_mgr.cpp:766) [862224][T1002_FreInfoRe][T1002][Y0-0000000000000000-0-0] [lt=109] ObTenantFreezeInfoMgr success to update infos(snapshot_gc_ts=1713924041263293000, min_major_snapshot=1, cur_idx=0, info_list_[0]=[{freeze_version:1, schema_version:1, cluster_version:17180000768}, {freeze_version:1713895203744844000, schema_version:1713890377976304, cluster_version:17180000768}], info_list_[1]=[{freeze_version:1, schema_version:1, cluster_version:17180000768}, {freeze_version:1713895203744844000, schema_version:1713890377976304, cluster_version:17180000768}], snapshots_[0]=[], snapshots_[1]=[]) [2024-04-24 10:00:51.385604] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=114] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50405}, multi_version_start=1713895420224546001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.386318] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=182] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:459}, multi_version_start=1713895420224546001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.387000] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=124] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60152}, multi_version_start=1713895421044983001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.387612] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60291}, multi_version_start=1713895421104689000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.388299] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=139] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60428}, multi_version_start=1713895421104689000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.389036] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50102}, multi_version_start=1713895421104689000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.389716] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=94] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:265}, multi_version_start=1713895421248988000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.390344] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60102}, multi_version_start=1713895421248988001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.391076] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50361}, multi_version_start=1713895421248988001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.391720] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=115] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:431}, multi_version_start=1713895421383659001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.392196] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:221) [861445][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=166] [ASYNC TASK QUEUE](queue_size=0) [2024-04-24 10:00:51.392328] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=129] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60295}, multi_version_start=1713895421383659001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.393025] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:112}, multi_version_start=1713895421487945001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.393637] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=113] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60241}, multi_version_start=1713895421487945001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.394404] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=112] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60359}, multi_version_start=1713895421487945001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.397240] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=106] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50297}, multi_version_start=1713895421647801000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.398108] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=115] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:437}, multi_version_start=1713895421647801000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.398772] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=128] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60365}, multi_version_start=1713895421651907002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.399544] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=150] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60208}, multi_version_start=1713895421651907002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.400400] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=150] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:158}, multi_version_start=1713895421814464001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.400955] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [861562][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=112] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2024-04-24 10:00:51.401103] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=119] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60434}, multi_version_start=1713895421852772000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.401810] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=116] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:400}, multi_version_start=1713895422051581000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.402588] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=144] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60348}, multi_version_start=1713895423020415000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.403215] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=97] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60169}, multi_version_start=1713895423020415002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.403910] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=125] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:322}, multi_version_start=1713895423020415002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.404617] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=124] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:345}, multi_version_start=1713895423337795000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.405238] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=120] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101042}, multi_version_start=1713895423337795000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.405807] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=160] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:145}, multi_version_start=1713895423337795000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.406482] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=152] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50328}, multi_version_start=1713895424391550000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.407141] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=78] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50344}, multi_version_start=1713895424391550000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.407747] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60317}, multi_version_start=1713895424600315001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.408342] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50224}, multi_version_start=1713895425164770000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.408593] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=122] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.408745] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=130] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.408831] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=74] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.408983] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=140] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:393}, multi_version_start=1713895425164770000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.409566] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=85] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:222}, multi_version_start=1713895425164770000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.410194] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=126] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60492}, multi_version_start=1713895425164770000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.410764] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=100] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60190}, multi_version_start=1713895425368363000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.411281] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=95] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50293}, multi_version_start=1713895425368363001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.411968] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:115}, multi_version_start=1713895425465714000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.412513] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50478}, multi_version_start=1713895426338214000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.413226] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=178] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60411}, multi_version_start=1713895426338214000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.413863] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60247}, multi_version_start=1713895426360834000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.414443] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=124] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:352}, multi_version_start=1713895426360834000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.415055] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [874613][T1003_Occam][T1003][YB420A000003-000616C2B70982E8-0-0] [lt=142] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:51.415115] INFO [SHARE.SCHEMA] fill_schema_operation (ob_schema_retrieve_utils.ipp:2698) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=108] fill schema operation(schema_operation=schema_version:1713924046573560, tenant_id:1004, database_id:500002, user_id:0, database_name:"", tablegroup_id:18446744073709551615, table_id:500926, operation_type:"OB_DDL_CREATE_TABLE", label_se_policy_id:500926, label_se_component_id:500926, label_se_label_id:500926, label_se_user_level_id:500926, outline_id:500926, udf_name:"", sequence_id:500926, keystore_id:500926, outline_id:500926, udf_name:"", sequence_id:500926, tablespace_id:500926, profile_id:500926, audit_id:500926, grantee_id:0, grantor_id:500002, ddl_stmt_str:"", dblink_id:500926, directory_id:500926) [2024-04-24 10:00:51.415272] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50476}, multi_version_start=1713895426538362000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.415291] INFO [SHARE.SCHEMA] fill_schema_operation (ob_schema_retrieve_utils.ipp:2698) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=111] fill schema operation(schema_operation=schema_version:1713924046573568, tenant_id:1004, database_id:500002, user_id:0, database_name:"", tablegroup_id:18446744073709551615, table_id:500927, operation_type:"OB_DDL_CREATE_INDEX", label_se_policy_id:500927, label_se_component_id:500927, label_se_label_id:500927, label_se_user_level_id:500927, outline_id:500927, udf_name:"", sequence_id:500927, keystore_id:500927, outline_id:500927, udf_name:"", sequence_id:500927, tablespace_id:500927, profile_id:500927, audit_id:500927, grantee_id:0, grantor_id:500002, ddl_stmt_str:"", dblink_id:500927, directory_id:500927) [2024-04-24 10:00:51.415328] WDIAG [OCCAM] dump_statistics (ob_vtable_event_recycle_buffer.h:144) [874613][T1003_Occam][T1003][YB420A000003-000616C2B70982E9-0-0] [lt=126][errcode=-4006] not init(ret=-4006) [2024-04-24 10:00:51.415390] INFO [SHARE.SCHEMA] fill_schema_operation (ob_schema_retrieve_utils.ipp:2698) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=76] fill schema operation(schema_operation=schema_version:1713924046573576, tenant_id:1004, database_id:500002, user_id:0, database_name:"", tablegroup_id:18446744073709551615, table_id:500926, operation_type:"OB_DDL_MODIFY_TABLE_SCHEMA_VERSION", label_se_policy_id:500926, label_se_component_id:500926, label_se_label_id:500926, label_se_user_level_id:500926, outline_id:500926, udf_name:"", sequence_id:500926, keystore_id:500926, outline_id:500926, udf_name:"", sequence_id:500926, tablespace_id:500926, profile_id:500926, audit_id:500926, grantee_id:0, grantor_id:500002, ddl_stmt_str:"", dblink_id:500926, directory_id:500926) [2024-04-24 10:00:51.415416] INFO [MDS] dump_map_holding_item (mds_tenant_service.cpp:556) [874613][T1003_Occam][T1003][YB420A000003-000616C2B70982E9-0-0] [lt=73] finish scan map holding items(scan_cnt=0) [2024-04-24 10:00:51.415478] INFO [SHARE.SCHEMA] fill_schema_operation (ob_schema_retrieve_utils.ipp:2698) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=64] fill schema operation(schema_operation=schema_version:1713924046573584, tenant_id:1004, database_id:0, user_id:0, database_name:"", tablegroup_id:0, table_id:0, operation_type:"OB_DDL_END_SIGN", label_se_policy_id:0, label_se_component_id:0, label_se_label_id:0, label_se_user_level_id:0, outline_id:0, udf_name:"", sequence_id:0, keystore_id:0, outline_id:0, udf_name:"", sequence_id:0, tablespace_id:0, profile_id:0, audit_id:0, grantee_id:0, grantor_id:0, ddl_stmt_str:"", dblink_id:0, directory_id:0) [2024-04-24 10:00:51.415914] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1713895200878015000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.416470] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=93] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:242}, multi_version_start=1713895426538362001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.417065] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=91] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50499}, multi_version_start=1713895426538362001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.417155] INFO [SHARE.SCHEMA] replay_log (ob_server_schema_service.cpp:4555) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=105] schema operation(schema_operation=schema_version:1713924046573560, tenant_id:1004, database_id:500002, user_id:0, database_name:"", tablegroup_id:18446744073709551615, table_id:500926, operation_type:"OB_DDL_CREATE_TABLE", label_se_policy_id:500926, label_se_component_id:500926, label_se_label_id:500926, label_se_user_level_id:500926, outline_id:500926, udf_name:"", sequence_id:500926, keystore_id:500926, outline_id:500926, udf_name:"", sequence_id:500926, tablespace_id:500926, profile_id:500926, audit_id:500926, grantee_id:0, grantor_id:500002, ddl_stmt_str:"", dblink_id:500926, directory_id:500926) [2024-04-24 10:00:51.417255] INFO [SHARE.SCHEMA] replay_log (ob_server_schema_service.cpp:4555) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=76] schema operation(schema_operation=schema_version:1713924046573568, tenant_id:1004, database_id:500002, user_id:0, database_name:"", tablegroup_id:18446744073709551615, table_id:500927, operation_type:"OB_DDL_CREATE_INDEX", label_se_policy_id:500927, label_se_component_id:500927, label_se_label_id:500927, label_se_user_level_id:500927, outline_id:500927, udf_name:"", sequence_id:500927, keystore_id:500927, outline_id:500927, udf_name:"", sequence_id:500927, tablespace_id:500927, profile_id:500927, audit_id:500927, grantee_id:0, grantor_id:500002, ddl_stmt_str:"", dblink_id:500927, directory_id:500927) [2024-04-24 10:00:51.417332] INFO [SHARE.SCHEMA] replay_log (ob_server_schema_service.cpp:4555) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=71] schema operation(schema_operation=schema_version:1713924046573576, tenant_id:1004, database_id:500002, user_id:0, database_name:"", tablegroup_id:18446744073709551615, table_id:500926, operation_type:"OB_DDL_MODIFY_TABLE_SCHEMA_VERSION", label_se_policy_id:500926, label_se_component_id:500926, label_se_label_id:500926, label_se_user_level_id:500926, outline_id:500926, udf_name:"", sequence_id:500926, keystore_id:500926, outline_id:500926, udf_name:"", sequence_id:500926, tablespace_id:500926, profile_id:500926, audit_id:500926, grantee_id:0, grantor_id:500002, ddl_stmt_str:"", dblink_id:500926, directory_id:500926) [2024-04-24 10:00:51.417396] INFO [SHARE.SCHEMA] replay_log (ob_server_schema_service.cpp:4555) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=57] schema operation(schema_operation=schema_version:1713924046573584, tenant_id:1004, database_id:0, user_id:0, database_name:"", tablegroup_id:0, table_id:0, operation_type:"OB_DDL_END_SIGN", label_se_policy_id:0, label_se_component_id:0, label_se_label_id:0, label_se_user_level_id:0, outline_id:0, udf_name:"", sequence_id:0, keystore_id:0, outline_id:0, udf_name:"", sequence_id:0, tablespace_id:0, profile_id:0, audit_id:0, grantee_id:0, grantor_id:0, ddl_stmt_str:"", dblink_id:0, directory_id:0) [2024-04-24 10:00:51.417513] INFO [SHARE.SCHEMA] add_tenant (ob_schema_mgr.cpp:1165) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=63] add tenant schema(ret=0, tenant_id=1004, tenant_schema={tenant_id:1004, schema_version:1713924046573584, tenant_name:"fake_tenant", name_case_mode:-1, read_only:false, primary_zone:"", locality:"", previous_locality:"", compatibility_mode:-1, gmt_modified:0, drop_tenant_time:0, status:0, in_recyclebin:false, arbitration_service_status:{status:3}}) [2024-04-24 10:00:51.417513] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:440}, multi_version_start=1713895426736541000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.417614] INFO [SHARE.SCHEMA] get_batch_users (ob_schema_service_sql_impl.cpp:3177) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=88] get batch users(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.417707] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3950) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=87] get batch users success(schema_keys=[]) [2024-04-24 10:00:51.417968] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [861702][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=178] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:51.418058] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=88] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50197}, multi_version_start=1713895426736541000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.417809] INFO [SHARE.SCHEMA] get_batch_databases (ob_schema_service_sql_impl.cpp:3178) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=92] get batch databases(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.418171] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3951) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=351] get batch databases success(schema_keys=[]) [2024-04-24 10:00:51.418244] INFO [SHARE.SCHEMA] get_batch_tablegroups (ob_schema_service_sql_impl.cpp:3179) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=64] get batch tablegroups(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.418317] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3952) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=64] get batch tablegroups success(schema_keys=[]) [2024-04-24 10:00:51.418398] INFO [SHARE.SCHEMA] get_batch_tables (ob_schema_service_sql_impl.cpp:3124) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=61] get batch tables(schema_version=1713924046573584, schema_keys=[{tenant_id:1004, user_id:18446744073709551615, database_id:18446744073709551615, tablegroup_id:18446744073709551615, table_id:500927, outline_id:500927, routine_name:"", routine_id:500927, database_name:"", table_name:"", schema_version:1713924046573568, synonym_id:500927, package_id:500927, trigger_id:500927, sequence_id:500927, sequence_name:"", udf_name:"", udt_id:500927, keystore_id:500927, keystore_name:"", label_se_policy_id:500927, label_se_component_id:500927, label_se_label_id:500927, label_se_user_level_id:500927, tablespace_id:500927, tablespace_name:"", profile_id:500927, audit_id:500927, grantee_id:18446744073709551615, grantor_id:18446744073709551615, col_id:18446744073709551615, obj_type:18446744073709551615, dblink_id:500927, directory_id:500927, context_id:500927, mock_fk_parent_table_id:500927, rls_policy_id:500927, rls_group_id:500927, rls_context_id:500927, routine_type:500927}, {tenant_id:1004, user_id:18446744073709551615, database_id:18446744073709551615, tablegroup_id:18446744073709551615, table_id:500926, outline_id:500926, routine_name:"", routine_id:500926, database_name:"", table_name:"", schema_version:1713924046573576, synonym_id:500926, package_id:500926, trigger_id:500926, sequence_id:500926, sequence_name:"", udf_name:"", udt_id:500926, keystore_id:500926, keystore_name:"", label_se_policy_id:500926, label_se_component_id:500926, label_se_label_id:500926, label_se_user_level_id:500926, tablespace_id:500926, tablespace_name:"", profile_id:500926, audit_id:500926, grantee_id:18446744073709551615, grantor_id:18446744073709551615, col_id:18446744073709551615, obj_type:18446744073709551615, dblink_id:500926, directory_id:500926, context_id:500926, mock_fk_parent_table_id:500926, rls_policy_id:500926, rls_group_id:500926, rls_context_id:500926, routine_type:500926}]) [2024-04-24 10:00:51.418438] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [874736][T1003_TenantWea][T1003][YB420A000003-000616C2B5FCA115-0-0] [lt=149] current clock is (clock_=0, max_seq_=0, mem_limit=264912214, cur_mem_hold=121634816, attr_.tenant_id_=1003) [2024-04-24 10:00:51.418551] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50351}, multi_version_start=1713895426776408001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.418926] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [861686][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=85] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1713924048840067000, v:0}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=2578851, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="10.0.0.2:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=437718, succ_cluster_heartbeat_count=437614, cluster_heartbeat_interval=50000, local_cluster_version={val:1713923469128896000, v:0}, local_cluster_delta=582290022, force_self_check=false, weak_read_refresh_interval=100000) [2024-04-24 10:00:51.419096] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=84] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50148}, multi_version_start=1713895426776408001, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.419715] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60222}, multi_version_start=1713895426943221000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.420030] INFO [SHARE.SCHEMA] retrieve_table_schema (ob_schema_retrieve_utils.ipp:107) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=101] retrieve table schema(ret=0, ret="OB_SUCCESS", table_schema={tenant_id:1004, database_id:500002, tablegroup_id:18446744073709551615, table_id:500927, association_table_id:18446744073709551615, in_offline_ddl_white_list:false, table_name:"__idx_500926_uk_merchantId_reportTime_channelId", session_id:0, index_type:2, table_type:5, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, tablespace_id:18446744073709551615, data_table_id:500926, name_casemode:-1, schema_version:1713924046573568, part_level:0, part_option:{part_func_type:0, part_func_expr:"", part_num:1, auto_part:false, auto_part_size:0}, sub_part_option:{part_func_type:0, part_func_expr:"", part_num:0, auto_part:false, auto_part_size:0}, partition_num:0, def_subpartition_num:0, partition_array:null, def_subpartition_array:null, hidden_partition_array:null, index_status:2, duplicate_scope:0, encryption:"", encrypt_key:"", master_key_id:18446744073709551615, sub_part_template_flags:0, get_tablet_id():{id:1152921504606847581}, max_dependency_version:-1, object_status:1, is_force_view:false, truncate_version:-1}, is_deleted=false) [2024-04-24 10:00:51.420180] INFO [SHARE.SCHEMA] retrieve_table_schema (ob_schema_retrieve_utils.ipp:107) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=102] retrieve table schema(ret=0, ret="OB_SUCCESS", table_schema={tenant_id:1004, database_id:500002, tablegroup_id:18446744073709551615, table_id:500926, association_table_id:18446744073709551615, in_offline_ddl_white_list:false, table_name:"gs_channel_roi_daily_stat", session_id:0, index_type:0, table_type:3, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, tablespace_id:18446744073709551615, data_table_id:0, name_casemode:-1, schema_version:1713924046573576, part_level:0, part_option:{part_func_type:0, part_func_expr:"", part_num:1, auto_part:false, auto_part_size:0}, sub_part_option:{part_func_type:0, part_func_expr:"", part_num:0, auto_part:false, auto_part_size:0}, partition_num:0, def_subpartition_num:0, partition_array:null, def_subpartition_array:null, hidden_partition_array:null, index_status:1, duplicate_scope:0, encryption:"", encrypt_key:"", master_key_id:18446744073709551615, sub_part_template_flags:0, get_tablet_id():{id:200317}, max_dependency_version:-1, object_status:1, is_force_view:false, truncate_version:-1}, is_deleted=false) [2024-04-24 10:00:51.420319] INFO [SHARE.SCHEMA] retrieve_table_schema (ob_schema_retrieve_utils.ipp:120) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=84] retrieve table schema array(tenant_id=1004) [2024-04-24 10:00:51.420487] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=143] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60290}, multi_version_start=1713895427004418000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.421011] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101038}, multi_version_start=1713895427004418000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.421500] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=101] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60260}, multi_version_start=1713895427138951000, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.421550] INFO [SHARE.SCHEMA] retrieve_simple_foreign_key_info (ob_schema_retrieve_utils.ipp:4794) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=67] retrieve simple foreign key info(tenant_id=1004) [2024-04-24 10:00:51.422050] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50169}, multi_version_start=1713895427138951002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.422524] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=110] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60238}, multi_version_start=1713895427138951002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.422747] INFO [SHARE.SCHEMA] retrieve_simple_constraint_info (ob_schema_retrieve_utils.ipp:4925) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=64] retrieve simple constraint info(tenant_id=1004) [2024-04-24 10:00:51.422985] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3953) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=80] get batch tables success(schema_keys=[{tenant_id:1004, user_id:18446744073709551615, database_id:18446744073709551615, tablegroup_id:18446744073709551615, table_id:500927, outline_id:500927, routine_name:"", routine_id:500927, database_name:"", table_name:"", schema_version:1713924046573568, synonym_id:500927, package_id:500927, trigger_id:500927, sequence_id:500927, sequence_name:"", udf_name:"", udt_id:500927, keystore_id:500927, keystore_name:"", label_se_policy_id:500927, label_se_component_id:500927, label_se_label_id:500927, label_se_user_level_id:500927, tablespace_id:500927, tablespace_name:"", profile_id:500927, audit_id:500927, grantee_id:18446744073709551615, grantor_id:18446744073709551615, col_id:18446744073709551615, obj_type:18446744073709551615, dblink_id:500927, directory_id:500927, context_id:500927, mock_fk_parent_table_id:500927, rls_policy_id:500927, rls_group_id:500927, rls_context_id:500927, routine_type:500927}, {tenant_id:1004, user_id:18446744073709551615, database_id:18446744073709551615, tablegroup_id:18446744073709551615, table_id:500926, outline_id:500926, routine_name:"", routine_id:500926, database_name:"", table_name:"", schema_version:1713924046573576, synonym_id:500926, package_id:500926, trigger_id:500926, sequence_id:500926, sequence_name:"", udf_name:"", udt_id:500926, keystore_id:500926, keystore_name:"", label_se_policy_id:500926, label_se_component_id:500926, label_se_label_id:500926, label_se_user_level_id:500926, tablespace_id:500926, tablespace_name:"", profile_id:500926, audit_id:500926, grantee_id:18446744073709551615, grantor_id:18446744073709551615, col_id:18446744073709551615, obj_type:18446744073709551615, dblink_id:500926, directory_id:500926, context_id:500926, mock_fk_parent_table_id:500926, rls_policy_id:500926, rls_group_id:500926, rls_context_id:500926, routine_type:500926}]) [2024-04-24 10:00:51.423064] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=86] no need to update min/max version(new_version={val:1713924048441114000, v:0}, min_version={val:1713924048441114000, v:0}, max_version={val:1713924048441114000, v:0}) [2024-04-24 10:00:51.423072] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101090}, multi_version_start=1713895427138951002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.423088] INFO [SHARE.SCHEMA] get_batch_outlines (ob_schema_service_sql_impl.cpp:3183) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=96] get batch outlines(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.423195] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3954) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=94] get batch outlines success(schema_keys=[]) [2024-04-24 10:00:51.423301] INFO [SHARE.SCHEMA] get_batch_routines (ob_schema_service_sql_impl.cpp:3185) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=97] get batch routines(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.423385] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3955) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=76] get batch routines success(schema_keys=[]) [2024-04-24 10:00:51.423475] INFO [SHARE.SCHEMA] get_batch_packages (ob_schema_service_sql_impl.cpp:3186) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=80] get batch packages(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.423562] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3956) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=81] get batch packages success(schema_keys=[]) [2024-04-24 10:00:51.423638] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60328}, multi_version_start=1713895427138951002, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.423686] INFO [SHARE.SCHEMA] get_batch_triggers (ob_schema_service_sql_impl.cpp:3187) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=114] get batch triggers(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.423765] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3957) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=71] get batch triggers success(schema_keys=[]) [2024-04-24 10:00:51.423843] INFO [SHARE.SCHEMA] get_batch_db_privs (ob_schema_service_sql_impl.cpp:3180) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=70] get batch db_privs(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.424005] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3958) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=155] get batch db_privs success(schema_keys=[]) [2024-04-24 10:00:51.424098] INFO [SHARE.SCHEMA] get_batch_table_privs (ob_schema_service_sql_impl.cpp:3181) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=79] get batch table_privs(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.424123] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:176) [861494][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=72] handle gts request statistics(total_rt=18265, total_cnt=592, avg_rt=3.080101180438448694e+01, avg_cnt=5.920000000000000284e+01) [2024-04-24 10:00:51.424206] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3959) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=99] get batch table_privs success(schema_keys=[]) [2024-04-24 10:00:51.424316] INFO [SHARE.SCHEMA] get_batch_routine_privs (ob_schema_service_sql_impl.cpp:3182) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=102] get batch routine_privs(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.424325] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=102] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:442}, multi_version_start=1713895427239649003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.424412] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3960) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=86] get batch routine_privs success(schema_keys=[]) [2024-04-24 10:00:51.424493] INFO [SHARE.SCHEMA] get_batch_synonyms (ob_schema_service_sql_impl.cpp:3184) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=73] get batch synonyms(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.424501] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=105] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1713924051424423], gts:1713924051423315000, latest_srr:[mts=1713924051424423]}) [2024-04-24 10:00:51.424582] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3961) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=79] get batch synonyms success(schema_keys=[]) [2024-04-24 10:00:51.424698] INFO [SHARE.SCHEMA] get_batch_udfs (ob_schema_service_sql_impl.cpp:3188) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=108] get batch udfs(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.424786] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3962) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=80] get batch udfs success(schema_keys=[]) [2024-04-24 10:00:51.424945] INFO [SHARE.SCHEMA] get_batch_udts (ob_schema_service_sql_impl.cpp:3189) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=150] get batch udts(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.424969] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=87] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:373}, multi_version_start=1713895427239649003, min_reserved_snapshot=1713922241269740000, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1713922241269740000, last_major_snapshot_version=1713895200575122000) [2024-04-24 10:00:51.425046] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3963) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=89] get batch udts success(schema_keys=[]) [2024-04-24 10:00:51.425071] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=100] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=393, get_gts_with_stc_cnt=35, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=28, try_wait_gts_elapse_cnt=0) [2024-04-24 10:00:51.425135] INFO [SHARE.SCHEMA] get_batch_sequences (ob_schema_service_sql_impl.cpp:3190) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=82] get batch sequences(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.425128] INFO [STORAGE] runTimerTask (ob_tenant_tablet_scheduler.cpp:174) [874751][T1003_SSTableGC][T1003][YB420A000003-000616C2B948F6D6-0-0] [lt=102] SSTableGCTask(cost_ts=786614) [2024-04-24 10:00:51.425279] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3964) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=131] get batch sequences success(schema_keys=[]) [2024-04-24 10:00:51.425405] INFO [SHARE.SCHEMA] get_batch_keystores (ob_schema_service_sql_impl.cpp:3191) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=113] get batch keystores(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.425492] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3965) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=76] get batch keystores success(schema_keys=[]) [2024-04-24 10:00:51.425599] INFO [SHARE.SCHEMA] get_batch_label_se_policys (ob_schema_service_sql_impl.cpp:3192) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=98] get batch label_se_policys(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.425693] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3966) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=82] get batch label_se_policys success(schema_keys=[]) [2024-04-24 10:00:51.425765] INFO [SHARE.SCHEMA] get_batch_label_se_components (ob_schema_service_sql_impl.cpp:3193) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=65] get batch label_se_components(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.425845] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3967) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=69] get batch label_se_components success(schema_keys=[]) [2024-04-24 10:00:51.425980] INFO [SHARE.SCHEMA] get_batch_label_se_labels (ob_schema_service_sql_impl.cpp:3194) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=124] get batch label_se_labels(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.426066] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3968) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=72] get batch label_se_labels success(schema_keys=[]) [2024-04-24 10:00:51.426167] INFO [SHARE.SCHEMA] get_batch_label_se_user_levels (ob_schema_service_sql_impl.cpp:3195) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=90] get batch label_se_user_levels(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.426293] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3969) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=113] get batch label_se_user_levels success(schema_keys=[]) [2024-04-24 10:00:51.426410] INFO [SHARE.SCHEMA] get_batch_tablespaces (ob_schema_service_sql_impl.cpp:3196) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=105] get batch tablespaces(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.426489] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3970) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=71] get batch tablespaces success(schema_keys=[]) [2024-04-24 10:00:51.426592] INFO [SHARE.SCHEMA] get_batch_profiles (ob_schema_service_sql_impl.cpp:3197) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=93] get batch profiles(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.426679] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3971) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=79] get batch profiles success(schema_keys=[]) [2024-04-24 10:00:51.426765] INFO [SHARE.SCHEMA] get_batch_audits (ob_schema_service_sql_impl.cpp:3198) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=71] get batch audits(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.426852] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3972) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=79] get batch audits success(schema_keys=[]) [2024-04-24 10:00:51.426977] INFO [SHARE.SCHEMA] get_batch_sys_privs (ob_schema_service_sql_impl.cpp:3199) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=118] get batch sys_privs(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.427080] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3973) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=89] get batch sys_privs success(schema_keys=[]) [2024-04-24 10:00:51.427170] INFO [SHARE.SCHEMA] get_batch_obj_privs (ob_schema_service_sql_impl.cpp:3200) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=78] get batch obj_privs(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.427258] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3974) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=81] get batch obj_privs success(schema_keys=[]) [2024-04-24 10:00:51.427397] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3981) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=130] get batch sys_variables success(schema_keys=[]) [2024-04-24 10:00:51.427487] INFO [SHARE.SCHEMA] get_batch_dblinks (ob_schema_service_sql_impl.cpp:3201) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=82] get batch dblinks(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.427602] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3982) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=105] get batch dblinks success(schema_keys=[]) [2024-04-24 10:00:51.427717] INFO [SHARE.SCHEMA] get_batch_directorys (ob_schema_service_sql_impl.cpp:3202) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=105] get batch directorys(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.427805] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3983) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=80] get batch directorys success(schema_keys=[]) [2024-04-24 10:00:51.427952] INFO [SHARE.SCHEMA] get_batch_contexts (ob_schema_service_sql_impl.cpp:3203) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=135] get batch contexts(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.428079] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3984) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=107] get batch contexts success(schema_keys=[]) [2024-04-24 10:00:51.428199] INFO [SHARE.SCHEMA] get_batch_mock_fk_parent_tables (ob_schema_service_sql_impl.cpp:3204) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=96] get batch mock_fk_parent_tables(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.428291] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3985) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=84] get batch mock_fk_parent_tables success(schema_keys=[]) [2024-04-24 10:00:51.428494] INFO [SHARE.SCHEMA] get_batch_rls_policys (ob_schema_service_sql_impl.cpp:3205) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=186] get batch rls_policys(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.428632] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3986) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=126] get batch rls_policys success(schema_keys=[]) [2024-04-24 10:00:51.428806] INFO [SHARE.SCHEMA] get_batch_rls_groups (ob_schema_service_sql_impl.cpp:3206) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=162] get batch rls_groups(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.428968] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3987) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=151] get batch rls_groups success(schema_keys=[]) [2024-04-24 10:00:51.429077] INFO [SHARE.SCHEMA] get_batch_rls_contexts (ob_schema_service_sql_impl.cpp:3207) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=103] get batch rls_contexts(schema_version=1713924046573584, schema_keys=[]) [2024-04-24 10:00:51.429197] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3988) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=109] get batch rls_contexts success(schema_keys=[]) [2024-04-24 10:00:51.429315] INFO [SHARE.SCHEMA] get_batch_table_schema (ob_schema_service_sql_impl.cpp:419) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=106] get batch table schema finish(schema_version=1713924046573584, ret=0) [2024-04-24 10:00:51.429467] INFO [SHARE.SCHEMA] apply_tenant_schema_to_cache (ob_server_schema_service.cpp:4164) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=98] del tenants finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.429590] INFO [SHARE.SCHEMA] apply_tenant_schema_to_cache (ob_server_schema_service.cpp:4207) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=105] add tenants finish(schemas=[], ret=0) [2024-04-24 10:00:51.429728] INFO [SHARE.SCHEMA] apply_tenant_schema_to_cache (ob_server_schema_service.cpp:4215) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=128] add tenants finish(schemas=[], ret=0) [2024-04-24 10:00:51.429853] INFO [SHARE.SCHEMA] apply_sys_variable_schema_to_cache (ob_server_schema_service.cpp:4260) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=111] del sys_variables finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.430022] INFO [SHARE.SCHEMA] apply_sys_variable_schema_to_cache (ob_server_schema_service.cpp:4260) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=154] add sys_variables finish(schemas=[], ret=0) [2024-04-24 10:00:51.430120] INFO [SHARE.SCHEMA] apply_user_schema_to_cache (ob_server_schema_service.cpp:4261) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=90] del users finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.430241] INFO [SHARE.SCHEMA] apply_user_schema_to_cache (ob_server_schema_service.cpp:4261) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=106] add users finish(schemas=[], ret=0) [2024-04-24 10:00:51.430328] INFO [SHARE.SCHEMA] apply_database_schema_to_cache (ob_server_schema_service.cpp:4262) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=75] del databases finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.430434] INFO [SHARE.SCHEMA] apply_database_schema_to_cache (ob_server_schema_service.cpp:4262) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=92] add databases finish(schemas=[], ret=0) [2024-04-24 10:00:51.430580] INFO [SHARE.SCHEMA] apply_tablegroup_schema_to_cache (ob_server_schema_service.cpp:4263) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=137] del tablegroups finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.430674] INFO [SHARE.SCHEMA] apply_tablegroup_schema_to_cache (ob_server_schema_service.cpp:4263) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=84] add tablegroups finish(schemas=[], ret=0) [2024-04-24 10:00:51.430779] INFO [SHARE.SCHEMA] apply_table_schema_to_cache (ob_server_schema_service.cpp:4264) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=99] del tables finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.431148] INFO [SHARE.SCHEMA] add_tables (ob_schema_mgr.cpp:2380) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=72] add tables(ret=0, ret="OB_SUCCESS", stage_cost=[24, 18, 16, 26, 26], cost=220) [2024-04-24 10:00:51.431259] INFO [SHARE.SCHEMA] apply_table_schema_to_cache (ob_server_schema_service.cpp:4264) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=105] add tables finish(schemas=[{tenant_id:1004, database_id:500002, tablegroup_id:18446744073709551615, table_id:500927, association_table_id:18446744073709551615, in_offline_ddl_white_list:false, table_name:"__idx_500926_uk_merchantId_reportTime_channelId", session_id:0, index_type:2, table_type:5, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, tablespace_id:18446744073709551615, data_table_id:500926, name_casemode:-1, schema_version:1713924046573568, part_level:0, part_option:{part_func_type:0, part_func_expr:"", part_num:1, auto_part:false, auto_part_size:0}, sub_part_option:{part_func_type:0, part_func_expr:"", part_num:0, auto_part:false, auto_part_size:0}, partition_num:0, def_subpartition_num:0, partition_array:null, def_subpartition_array:null, hidden_partition_array:null, index_status:2, duplicate_scope:0, encryption:"", encrypt_key:"", master_key_id:18446744073709551615, sub_part_template_flags:0, get_tablet_id():{id:1152921504606847581}, max_dependency_version:-1, object_status:1, is_force_view:false, truncate_version:-1}, {tenant_id:1004, database_id:500002, tablegroup_id:18446744073709551615, table_id:500926, association_table_id:18446744073709551615, in_offline_ddl_white_list:false, table_name:"gs_channel_roi_daily_stat", session_id:0, index_type:0, table_type:3, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, tablespace_id:18446744073709551615, data_table_id:0, name_casemode:-1, schema_version:1713924046573576, part_level:0, part_option:{part_func_type:0, part_func_expr:"", part_num:1, auto_part:false, auto_part_size:0}, sub_part_option:{part_func_type:0, part_func_expr:"", part_num:0, auto_part:false, auto_part_size:0}, partition_num:0, def_subpartition_num:0, partition_array:null, def_subpartition_array:null, hidden_partition_array:null, index_status:1, duplicate_scope:0, encryption:"", encrypt_key:"", master_key_id:18446744073709551615, sub_part_template_flags:0, get_tablet_id():{id:200317}, max_dependency_version:-1, object_status:1, is_force_view:false, truncate_version:-1}], ret=0) [2024-04-24 10:00:51.433829] INFO [SHARE.SCHEMA] apply_outline_schema_to_cache (ob_server_schema_service.cpp:4265) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=2553] del outlines finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.434045] INFO [SHARE.SCHEMA] apply_outline_schema_to_cache (ob_server_schema_service.cpp:4265) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=209] add outlines finish(schemas=[], ret=0) [2024-04-24 10:00:51.434125] INFO [SHARE.SCHEMA] apply_routine_schema_to_cache (ob_server_schema_service.cpp:4266) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=73] del routines finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.434215] INFO [SHARE.SCHEMA] apply_routine_schema_to_cache (ob_server_schema_service.cpp:4266) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=77] add routines finish(schemas=[], ret=0) [2024-04-24 10:00:51.434314] INFO [SHARE.SCHEMA] apply_package_schema_to_cache (ob_server_schema_service.cpp:4267) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=86] del packages finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.434400] INFO [SHARE.SCHEMA] apply_package_schema_to_cache (ob_server_schema_service.cpp:4267) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=75] add packages finish(schemas=[], ret=0) [2024-04-24 10:00:51.434389] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=80][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.434476] INFO [SHARE.SCHEMA] apply_trigger_schema_to_cache (ob_server_schema_service.cpp:4268) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=67] del triggers finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.434559] INFO [SHARE.SCHEMA] apply_trigger_schema_to_cache (ob_server_schema_service.cpp:4268) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=77] add triggers finish(schemas=[], ret=0) [2024-04-24 10:00:51.434708] INFO [SHARE.SCHEMA] apply_db_priv_schema_to_cache (ob_server_schema_service.cpp:4269) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=84] del db_privs finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.434797] INFO [SHARE.SCHEMA] apply_db_priv_schema_to_cache (ob_server_schema_service.cpp:4269) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=80] add db_privs finish(schemas=[], ret=0) [2024-04-24 10:00:51.434940] INFO [SHARE.SCHEMA] apply_table_priv_schema_to_cache (ob_server_schema_service.cpp:4270) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=133] del table_privs finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.435014] INFO [SHARE.SCHEMA] apply_table_priv_schema_to_cache (ob_server_schema_service.cpp:4270) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=68] add table_privs finish(schemas=[], ret=0) [2024-04-24 10:00:51.435092] INFO [SHARE.SCHEMA] apply_routine_priv_schema_to_cache (ob_server_schema_service.cpp:4271) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=70] del routine_privs finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.435157] INFO [SHARE.SCHEMA] apply_routine_priv_schema_to_cache (ob_server_schema_service.cpp:4271) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=58] add routine_privs finish(schemas=[], ret=0) [2024-04-24 10:00:51.435167] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=106] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.435264] INFO [SHARE.SCHEMA] apply_synonym_schema_to_cache (ob_server_schema_service.cpp:4272) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=97] del synonyms finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.435371] INFO [SHARE.SCHEMA] apply_synonym_schema_to_cache (ob_server_schema_service.cpp:4272) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=99] add synonyms finish(schemas=[], ret=0) [2024-04-24 10:00:51.435436] INFO [SHARE.SCHEMA] apply_udf_schema_to_cache (ob_server_schema_service.cpp:4273) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=53] del udfs finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.435504] INFO [SHARE.SCHEMA] apply_udf_schema_to_cache (ob_server_schema_service.cpp:4273) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=58] add udfs finish(schemas=[], ret=0) [2024-04-24 10:00:51.435578] INFO [SHARE.SCHEMA] apply_udt_schema_to_cache (ob_server_schema_service.cpp:4274) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=67] del udts finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.435646] INFO [SHARE.SCHEMA] apply_udt_schema_to_cache (ob_server_schema_service.cpp:4274) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=60] add udts finish(schemas=[], ret=0) [2024-04-24 10:00:51.435789] INFO [SHARE.SCHEMA] apply_sequence_schema_to_cache (ob_server_schema_service.cpp:4275) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=132] del sequences finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.435921] INFO [SHARE.SCHEMA] apply_sequence_schema_to_cache (ob_server_schema_service.cpp:4275) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=123] add sequences finish(schemas=[], ret=0) [2024-04-24 10:00:51.435994] INFO [SHARE.SCHEMA] apply_keystore_schema_to_cache (ob_server_schema_service.cpp:4276) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=67] del keystores finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.436076] INFO [SHARE.SCHEMA] apply_keystore_schema_to_cache (ob_server_schema_service.cpp:4276) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=71] add keystores finish(schemas=[], ret=0) [2024-04-24 10:00:51.436137] INFO [SHARE.SCHEMA] apply_label_se_policy_schema_to_cache (ob_server_schema_service.cpp:4277) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=55] del label_se_policys finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.436198] INFO [SHARE.SCHEMA] apply_label_se_policy_schema_to_cache (ob_server_schema_service.cpp:4277) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=53] add label_se_policys finish(schemas=[], ret=0) [2024-04-24 10:00:51.436267] INFO [SHARE.SCHEMA] apply_label_se_component_schema_to_cache (ob_server_schema_service.cpp:4278) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=57] del label_se_components finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.436346] INFO [SHARE.SCHEMA] apply_label_se_component_schema_to_cache (ob_server_schema_service.cpp:4278) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=71] add label_se_components finish(schemas=[], ret=0) [2024-04-24 10:00:51.436406] INFO [SHARE.SCHEMA] apply_label_se_label_schema_to_cache (ob_server_schema_service.cpp:4279) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=50] del label_se_labels finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.436493] INFO [SHARE.SCHEMA] apply_label_se_label_schema_to_cache (ob_server_schema_service.cpp:4279) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=74] add label_se_labels finish(schemas=[], ret=0) [2024-04-24 10:00:51.436581] INFO [SHARE.SCHEMA] apply_label_se_user_level_schema_to_cache (ob_server_schema_service.cpp:4280) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=79] del label_se_user_levels finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.436668] INFO [SHARE.SCHEMA] apply_label_se_user_level_schema_to_cache (ob_server_schema_service.cpp:4280) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=79] add label_se_user_levels finish(schemas=[], ret=0) [2024-04-24 10:00:51.436843] INFO [SHARE.SCHEMA] apply_tablespace_schema_to_cache (ob_server_schema_service.cpp:4281) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=167] del tablespaces finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.436984] INFO [SHARE.SCHEMA] apply_tablespace_schema_to_cache (ob_server_schema_service.cpp:4281) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=127] add tablespaces finish(schemas=[], ret=0) [2024-04-24 10:00:51.437063] INFO [SHARE.SCHEMA] apply_profile_schema_to_cache (ob_server_schema_service.cpp:4282) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=73] del profiles finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.437141] INFO [SHARE.SCHEMA] apply_profile_schema_to_cache (ob_server_schema_service.cpp:4282) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=64] add profiles finish(schemas=[], ret=0) [2024-04-24 10:00:51.437226] INFO [SHARE.SCHEMA] apply_audit_schema_to_cache (ob_server_schema_service.cpp:4283) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=78] del audits finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.437306] INFO [SHARE.SCHEMA] apply_audit_schema_to_cache (ob_server_schema_service.cpp:4283) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=72] add audits finish(schemas=[], ret=0) [2024-04-24 10:00:51.437368] INFO [SHARE.SCHEMA] apply_sys_priv_schema_to_cache (ob_server_schema_service.cpp:4284) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=55] del sys_privs finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.437441] INFO [SHARE.SCHEMA] apply_sys_priv_schema_to_cache (ob_server_schema_service.cpp:4284) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=61] add sys_privs finish(schemas=[], ret=0) [2024-04-24 10:00:51.437509] INFO [SHARE.SCHEMA] apply_obj_priv_schema_to_cache (ob_server_schema_service.cpp:4285) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=60] del obj_privs finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.437595] INFO [SHARE.SCHEMA] apply_obj_priv_schema_to_cache (ob_server_schema_service.cpp:4285) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=76] add obj_privs finish(schemas=[], ret=0) [2024-04-24 10:00:51.437691] INFO [SHARE.SCHEMA] apply_dblink_schema_to_cache (ob_server_schema_service.cpp:4286) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=79] del dblinks finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.437815] INFO [SHARE.SCHEMA] apply_dblink_schema_to_cache (ob_server_schema_service.cpp:4286) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=111] add dblinks finish(schemas=[], ret=0) [2024-04-24 10:00:51.437961] INFO [SHARE.SCHEMA] apply_directory_schema_to_cache (ob_server_schema_service.cpp:4287) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=132] del directorys finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.438037] INFO [SHARE.SCHEMA] apply_directory_schema_to_cache (ob_server_schema_service.cpp:4287) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=66] add directorys finish(schemas=[], ret=0) [2024-04-24 10:00:51.438117] INFO [SHARE.SCHEMA] apply_context_schema_to_cache (ob_server_schema_service.cpp:4288) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=73] del contexts finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.438194] INFO [SHARE.SCHEMA] apply_context_schema_to_cache (ob_server_schema_service.cpp:4288) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=62] add contexts finish(schemas=[], ret=0) [2024-04-24 10:00:51.438286] INFO [SHARE.SCHEMA] apply_mock_fk_parent_table_schema_to_cache (ob_server_schema_service.cpp:4289) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=83] del mock_fk_parent_tables finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.438349] INFO [SHARE.SCHEMA] apply_mock_fk_parent_table_schema_to_cache (ob_server_schema_service.cpp:4289) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=51] add mock_fk_parent_tables finish(schemas=[], ret=0) [2024-04-24 10:00:51.438411] INFO [SHARE.SCHEMA] apply_rls_policy_schema_to_cache (ob_server_schema_service.cpp:4290) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=56] del rls_policys finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.438496] INFO [SHARE.SCHEMA] apply_rls_policy_schema_to_cache (ob_server_schema_service.cpp:4290) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=75] add rls_policys finish(schemas=[], ret=0) [2024-04-24 10:00:51.438588] INFO [SHARE.SCHEMA] apply_rls_group_schema_to_cache (ob_server_schema_service.cpp:4291) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=82] del rls_groups finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.438733] INFO [SHARE.SCHEMA] apply_rls_group_schema_to_cache (ob_server_schema_service.cpp:4291) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=130] add rls_groups finish(schemas=[], ret=0) [2024-04-24 10:00:51.438855] INFO [SHARE.SCHEMA] apply_rls_context_schema_to_cache (ob_server_schema_service.cpp:4292) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=111] del rls_contexts finish(schema_keys=[], ret=0) [2024-04-24 10:00:51.439001] INFO [SHARE.SCHEMA] apply_rls_context_schema_to_cache (ob_server_schema_service.cpp:4292) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=133] add rls_contexts finish(schemas=[], ret=0) [2024-04-24 10:00:51.439150] INFO [SHARE.SCHEMA] add_schema (ob_multi_version_schema_service.cpp:2199) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=63] add schema(tenant_id=1004, refreshed_schema_version=1713924044295272, new_schema_version=1713924046573584) [2024-04-24 10:00:51.439295] INFO [SHARE.SCHEMA] alloc_ (ob_schema_mem_mgr.cpp:115) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=112] alloc schema mgr(tmp_ptr=0x14b04749a350, tmp_ptr=0x14b04749a350) [2024-04-24 10:00:51.439295] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0531-0-0] [lt=186][errcode=-4023] tx is 2pc logging(this={this:0x14b055e1a050, ref:5, trans_id:{txid:176911}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051285335], ctx_create_time:1713924048523054}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79627605}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051284482000, v:0}, trans_type:2, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051282008, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877467338312 trans_version={val:1713924051284482000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1 callback_alloc_count=31 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=56 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=56 unsynced_cnt=31 unsubmitted_cnt_=0 cb_statistics:[main=56, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051285620, block_frozen_memtable:null}) [2024-04-24 10:00:51.439489] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0531-0-0] [lt=184][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e1a050, ref:5, trans_id:{txid:176911}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051285335], ctx_create_time:1713924048523054}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79627605}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051284482000, v:0}, trans_type:2, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051282008, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877467338312 trans_version={val:1713924051284482000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1 callback_alloc_count=31 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=56 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=56 unsynced_cnt=31 unsubmitted_cnt_=0 cb_statistics:[main=56, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051285620, block_frozen_memtable:null}) [2024-04-24 10:00:51.439607] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0531-0-0] [lt=104][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176911}) [2024-04-24 10:00:51.439760] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0531-0-0] [lt=145][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176911}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:39, timestamp:1713924051425924, cluster_id:1713874979}, expire_ts:1713925048510047, commit_start_scn:{val:1713924051278217000, v:0}, parts:[{id:1}, {id:1001}], app_trace_info:""}) [2024-04-24 10:00:51.439952] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0531-0-0] [lt=178][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176911}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:39, timestamp:1713924051425924, cluster_id:1713874979}, expire_ts:1713925048510047, commit_start_scn:{val:1713924051278217000, v:0}, parts:[{id:1}, {id:1001}], app_trace_info:""}) [2024-04-24 10:00:51.440530] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:749) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=61] assign tenant_infos_ cost(ret=0, ret="OB_SUCCESS", cost=19) [2024-04-24 10:00:51.440722] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:756) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=66] assign user_infos_ cost(ret=0, ret="OB_SUCCESS", cost=73) [2024-04-24 10:00:51.440828] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:757) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=70] assign database_infos_ cost(ret=0, ret="OB_SUCCESS", cost=23) [2024-04-24 10:00:51.441005] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:758) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=139] assign database_name_map_ cost(ret=0, ret="OB_SUCCESS", cost=21) [2024-04-24 10:00:51.441110] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:759) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=69] assign tablegroup_infos_ cost(ret=0, ret="OB_SUCCESS", cost=19) [2024-04-24 10:00:51.441214] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:760) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=60] assign table_infos_ cost(ret=0, ret="OB_SUCCESS", cost=30) [2024-04-24 10:00:51.441324] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:761) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=65] assign index_infos_ cost(ret=0, ret="OB_SUCCESS", cost=31) [2024-04-24 10:00:51.441417] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:762) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=73] assign aux_vp_infos_ cost(ret=0, ret="OB_SUCCESS", cost=7) [2024-04-24 10:00:51.441514] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:763) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=62] assign lob_meta_infos_ cost(ret=0, ret="OB_SUCCESS", cost=21) [2024-04-24 10:00:51.441597] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:764) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=51] assign lob_piece_infos_ cost(ret=0, ret="OB_SUCCESS", cost=17) [2024-04-24 10:00:51.441690] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:765) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=51] assign drop_tenant_infos_ cost(ret=0, ret="OB_SUCCESS", cost=7) [2024-04-24 10:00:51.441803] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:632) [861663][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=88] finish to resize log external storage handler(current_ts=1713924051441797, tenant_max_cpu=3, valid_ls_count=0, other_ls_count=0, new_concurrency=0) [2024-04-24 10:00:51.441838] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:766) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=76] assign table_id_map_ cost(ret=0, ret="OB_SUCCESS", cost=49) [2024-04-24 10:00:51.442013] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:767) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=121] assign table_name_map_ cost(ret=0, ret="OB_SUCCESS", cost=39) [2024-04-24 10:00:51.442119] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:768) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=69] assign index_name_map_ cost(ret=0, ret="OB_SUCCESS", cost=23) [2024-04-24 10:00:51.442221] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:769) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=58] assign aux_vp_name_map_ cost(ret=0, ret="OB_SUCCESS", cost=28) [2024-04-24 10:00:51.442315] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:770) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=60] assign foreign_key_name_map_ cost(ret=0, ret="OB_SUCCESS", cost=20) [2024-04-24 10:00:51.442361] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1961) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=70] dump_dag_status(dag_cnt=0, map_size=0) [2024-04-24 10:00:51.442407] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:771) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=59] assign constraint_name_map_ cost(ret=0, ret="OB_SUCCESS", cost=20) [2024-04-24 10:00:51.442431] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1971) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(running_dag_net_map_size=0, blocking_dag_net_list_size=0) [2024-04-24 10:00:51.442510] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:772) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=59] assign hidden_table_name_map_ cost(ret=0, ret="OB_SUCCESS", cost=28) [2024-04-24 10:00:51.442531] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=94] dump_dag_status(priority="PRIO_COMPACTION_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:51.442591] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=54] dump_dag_status(priority="PRIO_HA_HIGH", low_limit=8, up_limit=8, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:51.442700] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=102] dump_dag_status(priority="PRIO_COMPACTION_MID", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:51.442779] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=70] dump_dag_status(priority="PRIO_HA_MID", low_limit=5, up_limit=5, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:51.442854] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=69] dump_dag_status(priority="PRIO_COMPACTION_LOW", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:51.442854] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=83] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-04-24 10:00:51.442993] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=129] dump_dag_status(priority="PRIO_HA_LOW", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:51.443192] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=186] dump_dag_status(priority="PRIO_DDL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:51.443360] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=58][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:1002197017}], scn_array=[{val:1713924050922254000, v:0}], curr_size=122, accum_size=122, time_cost=434336) [2024-04-24 10:00:51.443373] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=82][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179748652}], scn_array=[{val:1713924048988077001, v:0}], curr_size=1010, accum_size=1010, time_cost=335617) [2024-04-24 10:00:51.443414] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=187] dump_dag_status(priority="PRIO_DDL_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:51.443367] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=68][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:88127187}], scn_array=[{val:1713924051232826000, v:0}], curr_size=550, accum_size=4005, time_cost=209943) [2024-04-24 10:00:51.443483] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=74][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:298526301}], scn_array=[{val:1713924050970319001, v:0}], curr_size=121, accum_size=3120, time_cost=281525) [2024-04-24 10:00:51.443554] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=132] dump_dag_status(priority="PRIO_TTL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-04-24 10:00:51.443638] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=77] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:51.443714] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=87][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:314134313}], scn_array=[{val:1713924051049944000, v:0}], curr_size=542, accum_size=1150, time_cost=392259) [2024-04-24 10:00:51.443775] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=129] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:51.443809] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=73][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=673425, log_task={header:{begin_lsn:{lsn:1002197017}, end_lsn:{lsn:1002197139}, log_id:673425, min_scn:{val:1713924050922254000, v:0}, max_scn:{val:1713924050922254000, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1002197017}, data_checksum:2222604892, accum_checksum:1174158410, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924051008457, freeze_ts:1713924051008482, submit_ts:1713924051008746, flushed_ts:1713924051443695, gen_to_freeze cost time:25, gen_to_submit cost time:289, submit_to_flush cost time:434949}, fs_cb_begin_ts=1713924051443735, log_life_time=435278) [2024-04-24 10:00:51.443858] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=77] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:51.443911] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:826) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=54] ObSchemaMgr assign cost(ret=0, ret="OB_SUCCESS", cost=3368) [2024-04-24 10:00:51.444012] INFO [SHARE.SCHEMA] put (ob_schema_mgr_cache.cpp:502) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=95] put schema mgr(schema version=1713924046573584) [2024-04-24 10:00:51.444022] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=154] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:51.444048] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=88][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:165104275}], scn_array=[{val:1713924050962740000, v:0}], curr_size=1573, accum_size=1573, time_cost=322546) [2024-04-24 10:00:51.444100] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=71] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:51.444100] INFO [SHARE.SCHEMA] put (ob_schema_mgr_cache.cpp:576) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=74] dst schema mgr item ptr(tenant_id=1004, dst_item=0x14b11fbbe730, dst_timestamp=1713924051442693, dst_schema_version=1713924046573584, target_pos=7) [2024-04-24 10:00:51.444171] INFO [SHARE.SCHEMA] alloc_and_put_schema_mgr_ (ob_multi_version_schema_service.cpp:2281) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=57] put schema mgr succeed(schema_version=1713924046573584, eliminated_schema_version=-1, tenant_id=1004) [2024-04-24 10:00:51.444171] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=57] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:51.444150] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=49][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=505466, log_task={header:{begin_lsn:{lsn:314134313}, end_lsn:{lsn:314134855}, log_id:505466, min_scn:{val:1713924051049944000, v:0}, max_scn:{val:1713924051049944000, v:0}, data_len:486, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:314133705}, data_checksum:3697926766, accum_checksum:1528331468, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924051050991, freeze_ts:1713924051051014, submit_ts:1713924051051173, flushed_ts:1713924051443929, gen_to_freeze cost time:23, gen_to_submit cost time:182, submit_to_flush cost time:392756}, fs_cb_begin_ts=1713924051444113, log_life_time=393122) [2024-04-24 10:00:51.444194] INFO [LIB] stat (utility.h:1197) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=84] [PALF STAT LOG IO WORKER QUEUE SIZE](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1, this=0x14b1679f7068, extra_info="log_io_worker_num:4, cb_thread_pool_tg_id:395") [2024-04-24 10:00:51.444256] INFO [SHARE.SCHEMA] update_refreshed_version (ob_schema_store.cpp:60) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=79] [SCHEMA_STORE] schema store update version(tenant_id=1004, version=1713924046573584, refreshed_version=1713924046573584, received_version=1713924046573584) [2024-04-24 10:00:51.444256] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=79] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:51.444336] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=71] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:51.444336] INFO [SHARE.SCHEMA] add_schema (ob_multi_version_schema_service.cpp:2234) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=71] [REFRESH_SCHEMA] change refreshed_schema_version with new mode(tenant_id=1004, new_schema_version=1713924046573584) [2024-04-24 10:00:51.444488] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=143] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:51.444581] INFO [SHARE.SCHEMA] try_gc_allocator_when_add_schema_ (ob_multi_version_schema_service.cpp:3865) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=143] try to gc allocator's schema mgr which schema version is less than reserve_version(tenant_id=1004, eli_schema_version=1713924033807752, local_version=1713924046573584, reserve_version=1713924034253440) [2024-04-24 10:00:51.444591] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=75][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051049944000, v:0}, log_ts:{val:1713924051049944000, v:0}, lsn:{lsn:314134369}, submit_ts:1713924051049944}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568575}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=394631, ctx_lock_wait_time=6) [2024-04-24 10:00:51.444694] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=199] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:51.444814] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=109] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-04-24 10:00:51.444989] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=142] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-04-24 10:00:51.445073] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861506][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=74] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:51.445073] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=160][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67159711}], scn_array=[{val:1713924051045053000, v:0}], curr_size=498, accum_size=748, time_cost=193805) [2024-04-24 10:00:51.445079] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=84] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, dag_count=0) [2024-04-24 10:00:51.445134] INFO [SHARE.SCHEMA] free_schema_mgr (ob_schema_mem_mgr.cpp:260) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=120] [SCHEMA_RELEASE] free schema mgr(tenant_id=1004, schema_version=1713924033807752, timestamp_in_slot=1713924034801685) [2024-04-24 10:00:51.445178] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=73] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-04-24 10:00:51.445228] INFO [SHARE.SCHEMA] try_gc_allocator_when_add_schema_ (ob_multi_version_schema_service.cpp:3884) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=75] finish gc allocator's schema mgr when add schema(ret=0, ret="OB_SUCCESS", tenant_id=1004, total_schema_ptr_cnt=12, remain_schema_ptr_cnt=11, reserve_version=1713924034253440, cost_ts=733) [2024-04-24 10:00:51.445263] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=78] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0) [2024-04-24 10:00:51.445340] INFO [SHARE.SCHEMA] add_schema (ob_multi_version_schema_service.cpp:2241) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=61] finish add schema(ret=0, ret="OB_SUCCESS", tenant_id=1004, new_schema_version=1713924046573584, cost_ts=-6213) [2024-04-24 10:00:51.445384] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=114] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-04-24 10:00:51.445415] INFO [SHARE.SCHEMA] refresh_increment_schema (ob_server_schema_service.cpp:5856) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=62] change schema version(schema_status={tenant_id:1004, snapshot_timestamp:-1, readable_schema_version:-1}, schema_version=1713924046573584, core_schema_version=1713877451559520) [2024-04-24 10:00:51.445464] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=73] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0) [2024-04-24 10:00:51.445504] INFO [SHARE.SCHEMA] refresh_schema (ob_server_schema_service.cpp:5434) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=75] [REFRESH_SCHEMA] finish refresh increment schema(ret=0, schema_status={tenant_id:1004, snapshot_timestamp:-1, readable_schema_version:-1}, current schema_version=1713924046573584, cost=221257) [2024-04-24 10:00:51.445504] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861507][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=91] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:51.445523] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=53] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-04-24 10:00:51.445517] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3037) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B5CC83C0-0-0] [lt=80] start get_ls_replayed_scn(arg={tenant_id:1004, ls_id:{id:1002}}) [2024-04-24 10:00:51.445574] INFO [SHARE.SCHEMA] set_tenant_received_broadcast_version (ob_multi_version_schema_service.cpp:4100) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=57] try to set tenant received_broadcast_version(ret=0, tenant_id=1004, version=1713924046573584) [2024-04-24 10:00:51.445589] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=58] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0) [2024-04-24 10:00:51.445619] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3074) [2248569][T1003_L0_G0][T1004][YB420A000002-000616C2B5CC83C0-0-0] [lt=78] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1713924050784468000, v:0}, arg={tenant_id:1004, ls_id:{id:1002}}, result={tenant_id:1004, ls_id:{id:1002}, cur_readable_scn:{val:1713924050784468000, v:0}, offline_scn:{val:18446744073709551615, v:3}}, offline_scn={val:18446744073709551615, v:3}) [2024-04-24 10:00:51.445672] INFO [SHARE.SCHEMA] refresh_tenant_schema (ob_multi_version_schema_service.cpp:2889) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=72] [REFRESH_SCHEMA] end refresh and add schema by tenant(ret=0, ret="OB_SUCCESS", tenant_id=1004, cost=223415) [2024-04-24 10:00:51.445701] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=106] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-04-24 10:00:51.445769] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2572) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=80] [REFRESH_SCHEMA] end refresh and add schema(ret=0, ret="OB_SUCCESS", tenant_ids=[1004], cost=223644) [2024-04-24 10:00:51.445783] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=76] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:51.445836] INFO [SERVER] process_async_refresh_tasks (ob_server_schema_updater.cpp:405) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92487-0-0] [lt=60] try to async refresh schema(ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:51.445858] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=68] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:51.445946] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861509][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=98] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:51.445969] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=96] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:51.446144] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=144] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:51.446200] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861508][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=93] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:51.446230] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=75] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:51.446314] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=77] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:51.446379] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=58] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:51.446443] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=58] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:51.446515] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=60] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:51.446607] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=61] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:51.446738] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=125] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:51.446963] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=218] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:51.447044] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:51.447123] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=73] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:51.447224] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=89] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:51.447212] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C50-0-0] [lt=98][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=259321, log_task={header:{begin_lsn:{lsn:67159711}, end_lsn:{lsn:67160209}, log_id:259321, min_scn:{val:1713924051045053000, v:0}, max_scn:{val:1713924051045053000, v:0}, data_len:442, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:67159711}, data_checksum:2423324012, accum_checksum:1204586448, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924051046004, freeze_ts:1713924051250991, submit_ts:1713924051251143, flushed_ts:1713924051446372, gen_to_freeze cost time:204987, gen_to_submit cost time:205139, submit_to_flush cost time:195229}, fs_cb_begin_ts=1713924051447149, log_life_time=401145) [2024-04-24 10:00:51.447296] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=59] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:51.447307] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=104][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051045053000, v:0}, log_ts:{val:1713924051045053000, v:0}, lsn:{lsn:67159767}, submit_ts:1713924051045053}, this:0x14b168f7bb00, is_inited_:true, trans_id:{txid:869515}, ls_id:{id:1}, ctx:0x14b168f78b90, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=402214, ctx_lock_wait_time=29) [2024-04-24 10:00:51.447376] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:51.447495] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=109] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:51.447564] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=61] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:51.447638] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=66] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:51.447732] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=87] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:51.448003] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=259] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:51.448075] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861500][BatchIO][T0][Y0-0000000000000000-0-0] [lt=81] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:51.448003] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861511][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=147] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:51.448136] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=126] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:51.448227] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:51.448315] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=80] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:51.448389] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=68] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:51.448452] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=55] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:51.448527] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=69] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:51.448590] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=57] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-04-24 10:00:51.448698] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=101] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-04-24 10:00:51.448779] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-04-24 10:00:51.449007] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=219] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-04-24 10:00:51.449035] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861502][BatchIO][T0][Y0-0000000000000000-0-0] [lt=84] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:51.449048] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861501][BatchIO][T0][Y0-0000000000000000-0-0] [lt=94] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:51.449108] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=89] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-04-24 10:00:51.449226] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=108] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-04-24 10:00:51.449226] INFO [SHARE.SCHEMA] set_tenant_received_broadcast_version (ob_multi_version_schema_service.cpp:4100) [861736][T1_L0_G0][T1][YB420A000002-000616C2AD7C034D-0-0] [lt=119] try to set tenant received_broadcast_version(ret=0, tenant_id=1004, version=1713924046573584) [2024-04-24 10:00:51.449327] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=70] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:51.449344] INFO [SERVER] switch_schema (ob_service.cpp:1383) [861736][T1_L0_G0][T1][YB420A000002-000616C2AD7C034D-0-0] [lt=108] switch schema(ret=0, schema_info={schema_version:1713924046573584, tenant_id:1004, sequence_id:5497558139158}) [2024-04-24 10:00:51.449353] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3037) [862460][T1001_L0_G0][T1001][YB420A000002-000616C2AF5C88C9-0-0] [lt=78] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}}) [2024-04-24 10:00:51.449389] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861503][BatchIO][T0][Y0-0000000000000000-0-0] [lt=64] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:51.449426] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=83] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:51.449457] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3074) [862460][T1001_L0_G0][T1002][YB420A000002-000616C2AF5C88C9-0-0] [lt=89] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1713924048889699001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1713924048889699001, v:0}, offline_scn:{val:18446744073709551615, v:3}}, offline_scn={val:18446744073709551615, v:3}) [2024-04-24 10:00:51.449523] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=89] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:51.449592] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=61] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:51.449689] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=86] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:51.449777] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=80] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:51.449860] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=73] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:51.450062] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=166] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:51.450115] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861504][BatchIO][T0][Y0-0000000000000000-0-0] [lt=115] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:51.450145] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325892][T1_L0_G1][T1][YB420A000003-000616C2AA1E7FCB-0-0] [lt=74][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=364246, log_task={header:{begin_lsn:{lsn:179748652}, end_lsn:{lsn:179749662}, log_id:364246, min_scn:{val:1713924048988077001, v:0}, max_scn:{val:1713924051048470000, v:0}, data_len:954, proposal_id:5, prev_lsn:{lsn:179748114}, prev_proposal_id:5, committed_end_lsn:{lsn:179748652}, data_checksum:2220032083, accum_checksum:4038696542, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924051107128, freeze_ts:1713924051107135, submit_ts:1713924051107376, flushed_ts:1713924051443708, gen_to_freeze cost time:7, gen_to_submit cost time:248, submit_to_flush cost time:336332}, fs_cb_begin_ts=1713924051450088, log_life_time=342960) [2024-04-24 10:00:51.450145] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=72] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:51.450243] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=92] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:51.450348] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=98] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:51.450424] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=69] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:51.450516] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=80] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:51.450600] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:51.450689] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=80] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:51.450772] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861513][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=118] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:51.450772] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=72] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:51.450853] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-04-24 10:00:51.450937] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:142) [861721][T1_L0_G0][T1][YB420A000003-000616C2ABEB1E9C-0-0] [lt=101] transaction rpc statistics(total_rt=1089, total_process=15, avg_rt=68) [2024-04-24 10:00:51.451013] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=150] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-04-24 10:00:51.451096] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=75] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:51.451189] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=85] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:51.451241] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:943) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=451][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x14b11d0ac030, session_id:3221540160, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:206}, err:5, last_decode_time:1713924049409760, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1364, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:51.451265] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=70] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:51.451327] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=68] kill and revert session(conn.sessid_=3221540160, proxy_sessid=0, server_id=1, ret=0) [2024-04-24 10:00:51.451334] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:51.451419] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:918) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=85] sock ref clean, do destroy(*s={this:0x14b11d0ac030, session_id:3221540160, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:206}, err:5, last_decode_time:1713924049409760, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1364, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:51.451432] INFO [SHARE.SCHEMA] update_consensus_version (ob_schema_store.cpp:96) [861898][T1_L0_G0][T1][YB420A000002-000616C2AD7C034D-0-0] [lt=77] [SCHEMA_STORE] schema store update version(tenant_id=1004, version=1713924046573584, consensus_version=1713924046573584) [2024-04-24 10:00:51.451432] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=92] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:51.451511] INFO [SHARE.SCHEMA] set_tenant_broadcast_consensus_version (ob_multi_version_schema_service.cpp:4079) [861898][T1_L0_G0][T1][YB420A000002-000616C2AD7C034D-0-0] [lt=73] try to set tenant broadcast consensus version(ret=0, ret="OB_SUCCESS", tenant_id=1004, consensus_version=1713924046573584) [2024-04-24 10:00:51.451517] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=79] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:51.451523] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:159) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=72] unlock session of tenant(conn.sessid_=3221540160, proxy_sessid=0, conn.tenant_id_=1) [2024-04-24 10:00:51.451505] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=101] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.451576] WDIAG [COMMON] get (ob_link_hashmap.h:313) [861899][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=115][errcode=0] uref leak check(node->uref_=1073788695, lbt()="0x121f36ac 0x5547c4b 0xac68123 0xac69880 0x521a363 0x51fb0b7 0xa36ef16 0x4e1feed 0x4e0e175 0xa7ac354 0x12d1fdbc 0x14b1c1d132de 0x14b1c1a44e83") [2024-04-24 10:00:51.451589] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=66] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:51.451719] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=124] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:51.451610] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [861517][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=78] connection close(sessid=3221540160, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB420A000003-000616C2ABEB1E9C-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:124, pkt_rec[92]:{send:obp_mysql_header_{pkt_num_:141, seq_:12}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[93]:{send:obp_mysql_header_{len_:9, seq_:13}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[94]:{receive:obp_mysql_header_{len_:55, rec_:55, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[95]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[96]:{send:obp_mysql_header_{pkt_num_:90, seq_:18}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[97]:{send:obp_mysql_header_{len_:9, seq_:20}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[98]:{receive:obp_mysql_header_{len_:15, rec_:35, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[99]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[100]:{send:obp_mysql_header_{pkt_num_:16, seq_:3}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[101]:{send:obp_mysql_header_{len_:9, seq_:4}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[102]:{send:obp_mysql_header_{pkt_num_:64, seq_:12}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[103]:{send:obp_mysql_header_{len_:9, seq_:13}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[104]:{receive:obp_mysql_header_{len_:55, rec_:110, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[105]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[106]:{send:obp_mysql_header_{pkt_num_:71, seq_:18}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[107]:{send:obp_mysql_header_{len_:9, seq_:20}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[108]:{receive:obp_mysql_header_{len_:15, rec_:30, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[109]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[110]:{send:obp_mysql_header_{pkt_num_:16, seq_:3}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[111]:{send:obp_mysql_header_{len_:9, seq_:4}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[112]:{send:obp_mysql_header_{pkt_num_:64, seq_:12}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[113]:{send:obp_mysql_header_{len_:9, seq_:13}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[114]:{receive:obp_mysql_header_{len_:18, rec_:73, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[115]:{send:obp_mysql_header_{len_:11, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[116]:{receive:obp_mysql_header_{len_:133, rec_:203, seq_:0}, pkt_name:"COM_STMT_PREPARE", obp_mysql_header_.type_:22, obp_mysql_header_.is_send_:0}, pkt_rec[117]:{send:obp_mysql_header_{len_:16, seq_:2}, pkt_name:"PKT_PREPARE", obp_mysql_header_.is_send_:1}, pkt_rec[118]:{send:obp_mysql_header_{pkt_num_:38, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[119]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[120]:{receive:obp_mysql_header_{len_:74, rec_:89, seq_:0}, pkt_name:"COM_STMT_EXECUTE", obp_mysql_header_.type_:23, obp_mysql_header_.is_send_:0}, pkt_rec[121]:{send:obp_mysql_header_{len_:52, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[122]:{receive:obp_mysql_header_{len_:5, rec_:23, seq_:0}, pkt_name:"COM_STMT_CLOSE", obp_mysql_header_.type_:25, obp_mysql_header_.is_send_:0}, pkt_rec[123]:{send:obp_mysql_header_{len_:11, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2024-04-24 10:00:51.451745] INFO [SERVER] run (obmp_disconnect.cpp:82) [861899][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=70] free session successfully(sessid=3221540160) [2024-04-24 10:00:51.451834] INFO [SERVER] run (obmp_disconnect.cpp:89) [861899][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=79] mark session id unused(sessid=3221540160) [2024-04-24 10:00:51.451803] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=77] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:51.452042] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=227] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:51.452135] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=82] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:51.452228] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=83] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:51.452355] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=95] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:51.452445] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=82] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:51.452564] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=112] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:51.452678] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=107] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:51.452760] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=72] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:51.452842] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=72] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:51.452916] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:398) [861742][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=90] start to print blacklist info [2024-04-24 10:00:51.452973] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=123] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-04-24 10:00:51.453052] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=73] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-04-24 10:00:51.453139] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=81] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0) [2024-04-24 10:00:51.453222] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=76] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, scheduled_task_count=0) [2024-04-24 10:00:51.453332] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=103] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0) [2024-04-24 10:00:51.453393] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [861742][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=74] ls blacklist refresh finish(cost_time=536829) [2024-04-24 10:00:51.453409] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=70] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, scheduled_task_count=0) [2024-04-24 10:00:51.453496] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=81] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0) [2024-04-24 10:00:51.453577] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=75] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-04-24 10:00:51.453684] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=97] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0) [2024-04-24 10:00:51.453780] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=88] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-04-24 10:00:51.453940] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=154] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0) [2024-04-24 10:00:51.454032] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=85] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, scheduled_task_count=0) [2024-04-24 10:00:51.454127] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=87] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_net_count=0) [2024-04-24 10:00:51.454217] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=83] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_net_count=0) [2024-04-24 10:00:51.454336] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=111] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_net_count=0) [2024-04-24 10:00:51.454413] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=71] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_net_count=0) [2024-04-24 10:00:51.454491] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=66] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_net_count=0) [2024-04-24 10:00:51.454581] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=84] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_net_count=0) [2024-04-24 10:00:51.454675] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=81] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_net_count=0) [2024-04-24 10:00:51.454760] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=77] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_net_count=0) [2024-04-24 10:00:51.454951] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1991) [861645][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=180] dump_dag_status(total_worker_cnt=43, total_running_task_cnt=0, work_thread_num=43, scheduled_task_cnt=0) [2024-04-24 10:00:51.457968] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2C20C019D-0-0] [lt=84][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=301039, log_task={header:{begin_lsn:{lsn:165104275}, end_lsn:{lsn:165105848}, log_id:301039, min_scn:{val:1713924050962740000, v:0}, max_scn:{val:1713924050962740000, v:0}, data_len:1517, proposal_id:5, prev_lsn:{lsn:165104153}, prev_proposal_id:5, committed_end_lsn:{lsn:165104275}, data_checksum:1891944955, accum_checksum:4112143148, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924051120015, freeze_ts:1713924051120024, submit_ts:1713924051120123, flushed_ts:1713924051446264, gen_to_freeze cost time:9, gen_to_submit cost time:108, submit_to_flush cost time:326141}, fs_cb_begin_ts=1713924051457813, log_life_time=337798) [2024-04-24 10:00:51.460437] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=145] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.462813] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD2A1-0-0] [lt=72][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=330736, log_task={header:{begin_lsn:{lsn:88127187}, end_lsn:{lsn:88127737}, log_id:330736, min_scn:{val:1713924051232826000, v:0}, max_scn:{val:1713924051232826000, v:0}, data_len:494, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:88123732}, data_checksum:773904678, accum_checksum:2314739837, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924051233132, freeze_ts:1713924051233140, submit_ts:1713924051233282, flushed_ts:1713924051443735, gen_to_freeze cost time:8, gen_to_submit cost time:150, submit_to_flush cost time:210453}, fs_cb_begin_ts=1713924051462755, log_life_time=229623) [2024-04-24 10:00:51.462989] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2BFC8F2A9-0-0] [lt=89][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051232826000, v:0}, log_ts:{val:1713924051232826000, v:0}, lsn:{lsn:88127243}, submit_ts:1713924051232826}, this:0x14b1481bc200, is_inited_:true, trans_id:{txid:327206}, ls_id:{id:1001}, ctx:0x14b1481b9290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=230148, ctx_lock_wait_time=9) [2024-04-24 10:00:51.465318] INFO [STORAGE.TRANS] retransmit_downstream_msg_ (ob_two_phase_upstream_committer.cpp:311) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=102] unresponded participant(i=1, *this={this:0x14b055e1a050, ref:4, trans_id:{txid:176911}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051285335], ctx_create_time:1713924048523054}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79627605}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051284482000, v:0}, trans_type:2, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051282008, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877467338312 trans_version={val:1713924051284482000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1 callback_alloc_count=31 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=56 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=56 unsynced_cnt=31 unsubmitted_cnt_=0 cb_statistics:[main=56, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051285620, block_frozen_memtable:null}) [2024-04-24 10:00:51.465588] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=166] handle timeout(ret=0, *this={this:0x14b055e1a050, ref:5, trans_id:{txid:176911}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051285335], ctx_create_time:1713924048523054}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79627605}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051284482000, v:0}, trans_type:2, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051282008, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877467338312 trans_version={val:1713924051284482000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1 callback_alloc_count=31 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=56 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=56 unsynced_cnt=31 unsubmitted_cnt_=0 cb_statistics:[main=56, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051285620, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=100000) [2024-04-24 10:00:51.465826] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=227] trx is waiting log_cb(busy_cbs_.get_size()=2, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:0, v:0}, log_ts:{val:1713924051274688001, v:0}, lsn:{lsn:79627605}, submit_ts:1713924051284482}, this:0x14b055e1c8a8, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():4, range_array_:[{register_no_:1, has_submitted_:true, has_synced_:false, type:1, data_.length():76}, {register_no_:2, has_submitted_:true, has_synced_:false, type:1, data_.length():76}, {register_no_:3, has_submitted_:true, has_synced_:false, type:1, data_.length():67}, {register_no_:4, has_submitted_:true, has_synced_:false, type:6, data_.length():1099}]}, cb_arg_array_:[{log_type:0x4, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924051284482000, v:0}, log_ts:{val:1713924051284482000, v:0}, lsn:{lsn:79629066}, submit_ts:1713924051285620}, this:0x14b055e1caf0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x80, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:51.466067] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=163] handle timeout(ret=0, *this={this:0x14b055e2e050, ref:5, trans_id:{txid:176911}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924051092564}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88128808}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051278217001, v:0}, trans_type:2, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e30fc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1001}, ctx:0x14b055e2e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=0 trans_version={val:1713924051278217001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051182811, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=100000) [2024-04-24 10:00:51.466223] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=142] trx is waiting log_cb(busy_cbs_.get_size()=2, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:0, v:0}, log_ts:{val:1713924051274688001, v:0}, lsn:{lsn:88128808}, submit_ts:1713924051285620}, this:0x14b055e308a8, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1001}, ctx:0x14b055e2e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():2, range_array_:[{register_no_:1, has_submitted_:true, has_synced_:false, type:3, data_.length():159}, {register_no_:2, has_submitted_:true, has_synced_:false, type:1, data_.length():67}]}, cb_arg_array_:[{log_type:0x4, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924051278217001, v:0}, log_ts:{val:1713924051278217001, v:0}, lsn:{lsn:88129157}, submit_ts:1713924051286811}, this:0x14b055e30af0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1001}, ctx:0x14b055e2e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x40, arg:null}, {log_type:0x80, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:51.466366] INFO [STORAGE.TRANS] handle_2pc_prepare_request_impl_ (ob_two_phase_downstream_committer.cpp:335) [874587][T1004_L0_G0][T1004][Y0-0000000000000000-0-0] [lt=58] committer is under logging(ret=0, *this={this:0x14b055e2e050, ref:5, trans_id:{txid:176911}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924051092564}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88128808}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051278217001, v:0}, trans_type:2, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e30fc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1001}, ctx:0x14b055e2e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=0 trans_version={val:1713924051278217001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051182811, block_frozen_memtable:null}) [2024-04-24 10:00:51.467994] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=90][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.475592] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [2247256][T1001_L0_G0][T1001][YB420A000002-000616C2AF5C88C9-0-0] [lt=81] get number(ret=0, service_type_=0, range=1, base_id=1713924051474521000, start_id=1713924051474521000, end_id=1713924051474521001) [2024-04-24 10:00:51.481044] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=120] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.481141] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=77] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.484761] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [862433][T1001_Flush][T1001][Y0-0000000000000000-0-0] [lt=77] ====== traversal_flush timer task ====== [2024-04-24 10:00:51.485257] INFO [MDS] flush (mds_table_mgr.cpp:195) [862433][T1001_Flush][T1001][Y0-0000000000000000-0-0] [lt=88] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1713895313405863001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1713895313463635001, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1713895313529721001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1713895313529721002, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1713895313700741001, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1713895313700741003, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1713895313700741005, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1713895313903060001, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1713895313989809001, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1713895313989809002, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1713895314071277001, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1713895314071277003, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1713895314272137002, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1713895314312354001, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1713895314469076001, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1713895314547438001, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1713895314547438003, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1713895314689844001, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1713895314689844002, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1713895314783973001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1713895314815350001, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1713895314873839002, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1713895315201284002, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1713895315201284004, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1713895315742646002, v:0}}, {tablet_id:{id:60342}, rec_scn:{val:1713895315843910001, v:0}}, {tablet_id:{id:378}, rec_scn:{val:1713895315843910002, v:0}}, {tablet_id:{id:101036}, rec_scn:{val:1713895315944205001, v:0}}, {tablet_id:{id:60005}, rec_scn:{val:1713895315995531001, v:0}}, {tablet_id:{id:60130}, rec_scn:{val:1713895315995531003, v:0}}, {tablet_id:{id:101064}, rec_scn:{val:1713895316162213001, v:0}}, {tablet_id:{id:404}, rec_scn:{val:1713895316252755001, v:0}}, }, max_consequent_callbacked_scn={val:1713924048641803000, v:0}, *this={this:0x14b14c97a510, is_inited:true, freezing_scn:{val:1713877132047340000, v:0}, ls:{ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895203696037000, v:0}, clog_base_lsn:{lsn:134209536}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1713875407740689002, v:0}, all_id_meta:{id_meta:[{limited_id:1713924059338456000, latest_log_ts:{val:1713924042694950001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713898575276749001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1713910854505533001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:5, palf_env_:0x14b157ff2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:51.485428] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [862433][T1001_Flush][T1001][Y0-0000000000000000-0-0] [lt=152] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-04-24 10:00:51.508736] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=114] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.509194] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=84] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.509327] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=125] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.525164] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:834) [874425][T1004_BalanceEx][T1004][YB420A000003-000616C2BB69A619-0-0] [lt=84] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2024-04-24 10:00:51.525318] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:171) [874425][T1004_BalanceEx][T1004][YB420A000003-000616C2BB69A619-0-0] [lt=146] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2024-04-24 10:00:51.525417] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=84] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8901173248, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:51.534561] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=100][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.535349] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=97] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.549419] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [862260][T1002_TenantWea][T1001][YB420A000003-000616C2B18CC97C-0-0] [lt=72] current clock is (clock_=0, max_seq_=0, mem_limit=192099116, cur_mem_hold=130023424, attr_.tenant_id_=1001) [2024-04-24 10:00:51.549963] INFO [SHARE] get_balance_job (ob_balance_job_table_operator.cpp:209) [874429][T1004_TBalance][T1004][YB420A000003-000616C2BB790363-0-0] [lt=99] empty balance job(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=select time_to_usec(gmt_create) as start_time, time_to_usec(gmt_modified) as finish_time, * from __all_balance_job) [2024-04-24 10:00:51.551442] INFO [BALANCE] build (ob_transfer_partition_task.cpp:120) [874429][T1004_TBalance][T1004][YB420A000003-000616C2BB790363-0-0] [lt=87] [TRANSFER_PARTITION] no transfer partition task(tenant_id_=1004) [2024-04-24 10:00:51.551521] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:994) [874429][T1004_TBalance][T1004][YB420A000003-000616C2BB790363-0-0] [lt=63] no transfer partition job [2024-04-24 10:00:51.551593] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1024) [874429][T1004_TBalance][T1004][YB420A000003-000616C2BB790363-0-0] [lt=58] [TENANT_BALANCE] finish transfer partition(ret=0, ret="OB_SUCCESS", job_cnt=0, enable transfer={value_str:"True"}) [2024-04-24 10:00:51.551758] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=96] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.555215] INFO [BALANCE] gather_ls_status_stat (ob_tenant_balance_service.cpp:283) [874429][T1004_TBalance][T1004][YB420A000003-000616C2BB790363-0-0] [lt=119] [TENANT_BALANCE] LS no need balance(ls_status={tenant_id:1004, ls_id:{id:1}, ls_group_id:0, status:"NORMAL", unit_group_id:0, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2024-04-24 10:00:51.555417] INFO [BALANCE] ls_balance_ (ob_tenant_balance_service.cpp:481) [874429][T1004_TBalance][T1004][YB420A000003-000616C2BB790363-0-0] [lt=115] [TENANT_BALANCE] finish ls balance(ret=0, ret="OB_SUCCESS", need_ls_balance=false, job={tenant_id:0, job_id:{id:-1}, job_type:{val:-1, job_type:"INVALID"}, job_status:{val:-1, job_status:"INVALID"}, primary_zone_num:0, unit_group_num:0, comment:, balance_strategy:}, tasks=[]) [2024-04-24 10:00:51.555517] INFO [BALANCE] do_work (ob_tenant_balance_service.cpp:156) [874429][T1004_TBalance][T1004][YB420A000003-000616C2BB790363-0-0] [lt=78] [TENANT_BALANCE] finish one round(ret=0, ret="OB_SUCCESS", tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1004, job_cnt=0, primary_zone_num_=2, unit_group_array_=[{unit_group_id:1002, status:0}], ls_array_=[{tenant_id:1004, ls_id:{id:1001}, ls_group_id:1001, status:"NORMAL", unit_group_id:1002, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1004, ls_id:{id:1002}, ls_group_id:1001, status:"NORMAL", unit_group_id:1002, primary_zone:"zone2", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}], idle_time_us=10000000, last_partition_balance_time=1713920745455839, last_statistic_bg_stat_time=1713923781005624, last_statistic_schema_version=1713923779699968, last_statistic_max_transfer_task_id={id:-1}, enable_rebalance=true, enable_transfer=true) [2024-04-24 10:00:51.560598] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=107] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.564458] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [862259][T1002_ReqMemEvi][T1002][Y0-0000000000000000-0-0] [lt=147] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=16106127, evict_high_size_level=90000, evict_batch_count=0, elapse_time=9, size_used=6688, mem_used=12478464) [2024-04-24 10:00:51.568221] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=73][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.575844] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [862430][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=113] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=90000, evict_batch_count=0, elapse_time=9, size_used=8561, mem_used=14558208) [2024-04-24 10:00:51.581240] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=68] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.581369] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=122] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.608989] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=341] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.609113] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=105] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.609199] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=80] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.619416] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [861686][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=99] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-24 10:00:51.623566] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=89] no need to update min/max version(new_version={val:1713924048441114000, v:0}, min_version={val:1713924048441114000, v:0}, max_version={val:1713924048441114000, v:0}) [2024-04-24 10:00:51.625930] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [874532][T1004_LSArchive][T1004][YB420A000003-000616C2B568F0E7-0-0] [lt=93] gc stale ls task succ [2024-04-24 10:00:51.629117] INFO [SHARE] get_balance_job (ob_balance_job_table_operator.cpp:209) [862164][T1002_TBalance][T1002][YB420A000003-000616C2B3E90401-0-0] [lt=92] empty balance job(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=select time_to_usec(gmt_create) as start_time, time_to_usec(gmt_modified) as finish_time, * from __all_balance_job) [2024-04-24 10:00:51.630862] INFO [BALANCE] build (ob_transfer_partition_task.cpp:120) [862164][T1002_TBalance][T1002][YB420A000003-000616C2B3E90401-0-0] [lt=91] [TRANSFER_PARTITION] no transfer partition task(tenant_id_=1002) [2024-04-24 10:00:51.631286] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:994) [862164][T1002_TBalance][T1002][YB420A000003-000616C2B3E90401-0-0] [lt=416] no transfer partition job [2024-04-24 10:00:51.631370] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1024) [862164][T1002_TBalance][T1002][YB420A000003-000616C2B3E90401-0-0] [lt=77] [TENANT_BALANCE] finish transfer partition(ret=0, ret="OB_SUCCESS", job_cnt=0, enable transfer={value_str:"True"}) [2024-04-24 10:00:51.631540] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0532-0-0] [lt=65][errcode=-4023] tx is 2pc logging(this={this:0x14b055e1a050, ref:6, trans_id:{txid:176911}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051285335], ctx_create_time:1713924048523054}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79627605}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051284482000, v:0}, trans_type:2, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051282008, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877467338312 trans_version={val:1713924051284482000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1 callback_alloc_count=31 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=56 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=56 unsynced_cnt=31 unsubmitted_cnt_=0 cb_statistics:[main=56, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051285620, block_frozen_memtable:null}) [2024-04-24 10:00:51.631749] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0532-0-0] [lt=202][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e1a050, ref:6, trans_id:{txid:176911}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051285335], ctx_create_time:1713924048523054}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79627605}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051284482000, v:0}, trans_type:2, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051282008, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877467338312 trans_version={val:1713924051284482000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1 callback_alloc_count=31 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=56 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=56 unsynced_cnt=31 unsubmitted_cnt_=0 cb_statistics:[main=56, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051285620, block_frozen_memtable:null}) [2024-04-24 10:00:51.631976] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0532-0-0] [lt=217][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176911}) [2024-04-24 10:00:51.632058] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0532-0-0] [lt=76][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176911}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:39, timestamp:1713924051627068, cluster_id:1713874979}, expire_ts:1713925048510047, commit_start_scn:{val:1713924051278217000, v:0}, parts:[{id:1}, {id:1001}], app_trace_info:""}) [2024-04-24 10:00:51.632141] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0532-0-0] [lt=67][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176911}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:39, timestamp:1713924051627068, cluster_id:1713874979}, expire_ts:1713925048510047, commit_start_scn:{val:1713924051278217000, v:0}, parts:[{id:1}, {id:1001}], app_trace_info:""}) [2024-04-24 10:00:51.632751] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [862265][T1002_TabletShe][T1002][Y0-0000000000000000-0-0] [lt=97] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-04-24 10:00:51.632947] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [862265][T1002_TabletShe][T1002][Y0-0000000000000000-0-0] [lt=98] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=3, times=9730) [2024-04-24 10:00:51.634751] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=87][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.634728] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [874613][T1003_Occam][T1003][YB420A000003-000616C2B70982EA-0-0] [lt=62] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:51.635068] INFO [BALANCE] gather_ls_status_stat (ob_tenant_balance_service.cpp:283) [862164][T1002_TBalance][T1002][YB420A000003-000616C2B3E90401-0-0] [lt=91] [TENANT_BALANCE] LS no need balance(ls_status={tenant_id:1002, ls_id:{id:1}, ls_group_id:0, status:"NORMAL", unit_group_id:0, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2024-04-24 10:00:51.635263] INFO [BALANCE] ls_balance_ (ob_tenant_balance_service.cpp:481) [862164][T1002_TBalance][T1002][YB420A000003-000616C2B3E90401-0-0] [lt=113] [TENANT_BALANCE] finish ls balance(ret=0, ret="OB_SUCCESS", need_ls_balance=false, job={tenant_id:0, job_id:{id:-1}, job_type:{val:-1, job_type:"INVALID"}, job_status:{val:-1, job_status:"INVALID"}, primary_zone_num:0, unit_group_num:0, comment:, balance_strategy:}, tasks=[]) [2024-04-24 10:00:51.635545] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=78] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.638201] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:508) [862164][T1002_TBalance][T1002][YB420A000003-000616C2B3E90401-0-0] [lt=91][errcode=-4152] get obj error(ret=-4152) [2024-04-24 10:00:51.638398] INFO [BALANCE] do_work (ob_tenant_balance_service.cpp:156) [862164][T1002_TBalance][T1002][YB420A000003-000616C2B3E90401-0-0] [lt=80] [TENANT_BALANCE] finish one round(ret=0, ret="OB_SUCCESS", tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, job_cnt=0, primary_zone_num_=2, unit_group_array_=[{unit_group_id:1001, status:0}], ls_array_=[{tenant_id:1002, ls_id:{id:1001}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1002}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone2", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}], idle_time_us=10000000, last_partition_balance_time=1713918810622970, last_statistic_bg_stat_time=1713890966527040, last_statistic_schema_version=1713890377976304, last_statistic_max_transfer_task_id={id:-1}, enable_rebalance=true, enable_transfer=true) [2024-04-24 10:00:51.644417] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=129] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1006592, replace_num=15728) [2024-04-24 10:00:51.644505] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=77] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-04-24 10:00:51.646041] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [861685][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=98] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=11, size_used=15998, mem_used=27036672) [2024-04-24 10:00:51.651935] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=163] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.660933] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=79] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.667852] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8EC2750-0-0] [lt=91][errcode=-4023] tx is 2pc logging(this={this:0x14b0fc0a2b50, ref:3, trans_id:{txid:338715}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924081443033, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051470854], ctx_create_time:1713924051456595}{ls_id:{id:1}, session_id:3221795785, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051465459, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924051470594000, v:0}, log_ts:{val:1713924051470594000, v:0}, lsn:{lsn:298527506}, submit_ts:1713924051470594}, this:0x14b0fc0a5ac0, is_inited_:true, trans_id:{txid:338715}, ls_id:{id:1}, ctx:0x14b0fc0a2b50, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338715}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924051470594000, v:0}, start_scn:{val:1713924051470594000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924051470594000, v:0} commit_version={val:1713924051470594000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338715} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3688062677 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924051470594000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051470594, block_frozen_memtable:null}) [2024-04-24 10:00:51.668254] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8EC2750-0-0] [lt=392][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fc0a2b50, ref:3, trans_id:{txid:338715}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924081443033, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051470854], ctx_create_time:1713924051456595}{ls_id:{id:1}, session_id:3221795785, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051465459, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924051470594000, v:0}, log_ts:{val:1713924051470594000, v:0}, lsn:{lsn:298527506}, submit_ts:1713924051470594}, this:0x14b0fc0a5ac0, is_inited_:true, trans_id:{txid:338715}, ls_id:{id:1}, ctx:0x14b0fc0a2b50, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338715}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924051470594000, v:0}, start_scn:{val:1713924051470594000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924051470594000, v:0} commit_version={val:1713924051470594000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338715} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3688062677 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924051470594000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051470594, block_frozen_memtable:null}) [2024-04-24 10:00:51.668397] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=93][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.668405] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8EC2750-0-0] [lt=143][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338715}) [2024-04-24 10:00:51.668480] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8EC2750-0-0] [lt=66][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338715}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924051663263, cluster_id:1713874979}, expire_ts:1713924081466140, commit_start_scn:{val:1713924051458925000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:51.668583] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8EC2750-0-0] [lt=85][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338715}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924051663263, cluster_id:1713874979}, expire_ts:1713924081466140, commit_start_scn:{val:1713924051458925000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:51.671292] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [861673][T1_LSMetaCh][T1][YB420A000003-000616C2A909ADA5-0-0] [lt=174] fetch root ls success(local_cluster_id=1713874979, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.2:2882", sql_port:2881, role:1, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:5, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713921656566585, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.3:2882", sql_port:2881, role:2, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["10.0.0.2:2882", "10.0.0.3:2882"]) [2024-04-24 10:00:51.681450] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=95] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.681574] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=113] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.704394] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [862460][T1001_L0_G0][T1001][YB420A000002-000616C2B27C260B-0-0] [lt=72][errcode=-4023] tx is 2pc logging(this={this:0x14b15a15d590, ref:3, trans_id:{txid:391373}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924081449792, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051475560], ctx_create_time:1713924051461248}{ls_id:{id:1}, session_id:3221795795, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051472069, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924051474521000, v:0}, log_ts:{val:1713924051474521000, v:0}, lsn:{lsn:314135992}, submit_ts:1713924051474521}, this:0x14b15a160500, is_inited_:true, trans_id:{txid:391373}, ls_id:{id:1}, ctx:0x14b15a15d590, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:391373}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924051474521000, v:0}, start_scn:{val:1713924051474521000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875442714112 max_table_version=1713875442714112 trans_version={val:1713924051474521000, v:0} commit_version={val:1713924051474521000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:391373} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=2895691470 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924051474521000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051475837, block_frozen_memtable:null}) [2024-04-24 10:00:51.704595] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [862460][T1001_L0_G0][T1001][YB420A000002-000616C2B27C260B-0-0] [lt=188][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b15a15d590, ref:3, trans_id:{txid:391373}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924081449792, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051475560], ctx_create_time:1713924051461248}{ls_id:{id:1}, session_id:3221795795, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051472069, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924051474521000, v:0}, log_ts:{val:1713924051474521000, v:0}, lsn:{lsn:314135992}, submit_ts:1713924051474521}, this:0x14b15a160500, is_inited_:true, trans_id:{txid:391373}, ls_id:{id:1}, ctx:0x14b15a15d590, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:391373}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924051474521000, v:0}, start_scn:{val:1713924051474521000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875442714112 max_table_version=1713875442714112 trans_version={val:1713924051474521000, v:0} commit_version={val:1713924051474521000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:391373} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=2895691470 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924051474521000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051475837, block_frozen_memtable:null}) [2024-04-24 10:00:51.704808] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [862460][T1001_L0_G0][T1001][YB420A000002-000616C2B27C260B-0-0] [lt=204][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:391373}) [2024-04-24 10:00:51.704959] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [862460][T1001_L0_G0][T1001][YB420A000002-000616C2B27C260B-0-0] [lt=139][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1001, tx_id:{txid:391373}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924051699657, cluster_id:1713874979}, expire_ts:1713924081470822, commit_start_scn:{val:1713924051462734001, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:51.705085] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [862460][T1001_L0_G0][T1001][YB420A000002-000616C2B27C260B-0-0] [lt=105][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1001, tx_id:{txid:391373}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924051699657, cluster_id:1713874979}, expire_ts:1713924081470822, commit_start_scn:{val:1713924051462734001, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:51.709189] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=115] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.709307] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=101] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.709393] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=79] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.715092] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [874542][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=134] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1001}, {id:1}]) [2024-04-24 10:00:51.718001] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [862276][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=82] start do ls ha handler(ls_id_array_=[{id:1}, {id:1002}, {id:1001}]) [2024-04-24 10:00:51.725018] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=141] get wrs ts(ls_id={id:1}, delta=3082223, log_service_ts=1713924048641803, min_tx_service_ts=4611686018427387, timestamp={val:1713924048641803000, v:0}) [2024-04-24 10:00:51.725143] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=117] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=36, Other_Error_Cnt=0, Submit_Succ_Cnt=15, last_scn="{val:1713924029855439000, v:0}", last_lsn={lsn:314049889}, last_gts={val:1713924051114377000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-24 10:00:51.726242] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=96] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8903270400, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:51.735090] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=66][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.735731] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=76] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.738161] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [861547][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=71] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3}}, mode:0, create_timestamp:1713875175740107, is_removed:false}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:false, version:2}, create_status:1}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:53, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:16820896499, recv_hp_rpc_cnt:43602, recv_np_rpc_cnt:17741, recv_lp_rpc_cnt:0, recv_mysql_cnt:271304, recv_task_cnt:46871, recv_large_req_cnt:0, tt_large_quries:23685, pop_normal_cnt:141782315, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=1 cnt[3]=0 cnt[4]=0 cnt[5]=41822 cnt[6]=9 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:group_id = 2,queue_size = 0,recv_req_cnt = 97653,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1713875179031712 group_id = 10,queue_size = 0,recv_req_cnt = 97300,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1713875182851345 group_id = 15,queue_size = 0,recv_req_cnt = 138,min_worker_cnt = 2,max_worker_cnt = 2,worker_cnt = 0,token_change = 1713918443174315 group_id = 5,queue_size = 0,recv_req_cnt = 106,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1713875182021644 group_id = 100,queue_size = 0,recv_req_cnt = 456,min_worker_cnt = 8,max_worker_cnt = 150,worker_cnt = 0,token_change = 1713921726340253 group_id = 9,queue_size = 0,recv_req_cnt = 64885,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1713875168519697 group_id = 1,queue_size = 0,recv_req_cnt = 502191,min_worker_cnt = 8,max_worker_cnt = 8,worker_cnt = 1,token_change = 1713924014884373 , rpc_stat_info: pcode=0x1501:cnt=81 pcode=0x1516:cnt=27 pcode=0x150a:cnt=20 pcode=0x4a9:cnt=20 pcode=0x651:cnt=15, token_change_ts:1713875333439184, tenant_role:1}) [2024-04-24 10:00:51.739170] INFO do_work (ob_rl_mgr.cpp:709) [861413][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=95] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-24 10:00:51.738830] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [861547][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=662] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:5, max_cpu:5, memory_size:"1GB", log_disk_size:"2GB", min_iops:10000, max_iops:50000, iops_weight:0}}, mode:0, create_timestamp:1713875166997627, is_removed:false}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:true, version:2}, create_status:1}, unit_min_cpu:"5.000000000000000000e+00", unit_max_cpu:"5.000000000000000000e+00", total_worker_cnt:52, min_worker_cnt:52, max_worker_cnt:150, stopped:0, worker_us:9041140381, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:96404, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:236418280, workers:52, nesting workers:0, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:group_id = 15,queue_size = 0,recv_req_cnt = 1256,min_worker_cnt = 2,max_worker_cnt = 2,worker_cnt = 0,token_change = 1713918443597331 , rpc_stat_info: pcode=0x5c0:cnt=30, token_change_ts:1713875333456570, tenant_role:0}) [2024-04-24 10:00:51.739504] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [861547][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=662] dump tenant info(tenant={id:1001, tenant_meta:{unit:{tenant_id:1001, unit_id:1002, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"ocp_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.6GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:1}}, mode:0, create_timestamp:1713875391883804, is_removed:false}, super_block:{tenant_id:1001, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:false, version:2}, create_status:1}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:10, min_worker_cnt:6, max_worker_cnt:150, stopped:0, worker_us:9236382206, recv_hp_rpc_cnt:92009, recv_np_rpc_cnt:490342, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:19104826, workers:6, nesting workers:0, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=9 cnt[3]=0 cnt[4]=0 cnt[5]=658686 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:group_id = 2,queue_size = 0,recv_req_cnt = 291675,min_worker_cnt = 1,max_worker_cnt = 150,worker_cnt = 1,token_change = 1713875397107009 group_id = 5,queue_size = 0,recv_req_cnt = 219518,min_worker_cnt = 1,max_worker_cnt = 150,worker_cnt = 1,token_change = 1713875399979376 group_id = 19,queue_size = 0,recv_req_cnt = 24807,min_worker_cnt = 1,max_worker_cnt = 150,worker_cnt = 1,token_change = 1713923998246158 group_id = 1,queue_size = 0,recv_req_cnt = 535534,min_worker_cnt = 8,max_worker_cnt = 8,worker_cnt = 1,token_change = 1713924048243850 , rpc_stat_info: pcode=0x51c:cnt=126 pcode=0x717:cnt=91 pcode=0x1502:cnt=62 pcode=0x710:cnt=40 pcode=0x150b:cnt=40, token_change_ts:1713921649200845, tenant_role:1}) [2024-04-24 10:00:51.740261] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [861547][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=750] dump tenant info(tenant={id:1002, tenant_meta:{unit:{tenant_id:1002, unit_id:1002, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"ocp_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"5.4GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:1}}, mode:0, create_timestamp:1713875391883804, is_removed:false}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=1, log_id=100011, offset=22619338}, ls_meta_entry:[2877](ver=0,mode=0,seq=3431), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:false, version:2}, create_status:1}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:18, min_worker_cnt:6, max_worker_cnt:150, stopped:0, worker_us:23106469972, recv_hp_rpc_cnt:920057, recv_np_rpc_cnt:2432141, recv_lp_rpc_cnt:0, recv_mysql_cnt:423784, recv_task_cnt:606, recv_large_req_cnt:0, tt_large_quries:194552, pop_normal_cnt:23757153, workers:6, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=22 cnt[3]=0 cnt[4]=0 cnt[5]=312822 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:group_id = 2,queue_size = 0,recv_req_cnt = 678505,min_worker_cnt = 1,max_worker_cnt = 150,worker_cnt = 1,token_change = 1713875493604067 group_id = 15,queue_size = 0,recv_req_cnt = 184,min_worker_cnt = 2,max_worker_cnt = 2,worker_cnt = 0,token_change = 1713918461304992 group_id = 5,queue_size = 0,recv_req_cnt = 698678,min_worker_cnt = 1,max_worker_cnt = 150,worker_cnt = 1,token_change = 1713875496842494 group_id = 19,queue_size = 0,recv_req_cnt = 5151,min_worker_cnt = 1,max_worker_cnt = 150,worker_cnt = 1,token_change = 1713923997875683 group_id = 100,queue_size = 0,recv_req_cnt = 95,min_worker_cnt = 8,max_worker_cnt = 150,worker_cnt = 0,token_change = 1713921717966163 group_id = 1,queue_size = 0,recv_req_cnt = 1433223,min_worker_cnt = 8,max_worker_cnt = 8,worker_cnt = 1,token_change = 1713924042727409 , rpc_stat_info: pcode=0x52b:cnt=384 pcode=0x710:cnt=175 pcode=0x1502:cnt=140 pcode=0x650:cnt=122 pcode=0x659:cnt=98, token_change_ts:1713921614611023, tenant_role:1}) [2024-04-24 10:00:51.740953] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [861547][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=683] dump tenant info(tenant={id:1003, tenant_meta:{unit:{tenant_id:1003, unit_id:1003, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1004, name:"c8", resource:{min_cpu:1, max_cpu:1, memory_size:"1.2GB", log_disk_size:"3.6GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:2}}, mode:0, create_timestamp:1713877325237012, is_removed:false}, super_block:{tenant_id:1003, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:false, version:2}, create_status:1}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:10, min_worker_cnt:6, max_worker_cnt:150, stopped:0, worker_us:8100212830, recv_hp_rpc_cnt:89502, recv_np_rpc_cnt:487052, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:18449544, workers:6, nesting workers:0, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=1464 cnt[3]=0 cnt[4]=0 cnt[5]=629633 cnt[6]=2 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:group_id = 2,queue_size = 0,recv_req_cnt = 280075,min_worker_cnt = 1,max_worker_cnt = 150,worker_cnt = 1,token_change = 1713877331109404 group_id = 5,queue_size = 0,recv_req_cnt = 211923,min_worker_cnt = 1,max_worker_cnt = 150,worker_cnt = 1,token_change = 1713877335036739 group_id = 19,queue_size = 0,recv_req_cnt = 22981,min_worker_cnt = 1,max_worker_cnt = 150,worker_cnt = 1,token_change = 1713923998166576 group_id = 1,queue_size = 0,recv_req_cnt = 511557,min_worker_cnt = 8,max_worker_cnt = 8,worker_cnt = 1,token_change = 1713924041501828 , rpc_stat_info: pcode=0x51c:cnt=356 pcode=0x717:cnt=89 pcode=0x1502:cnt=65 pcode=0x710:cnt=49 pcode=0x150b:cnt=40, token_change_ts:1713921650829474, tenant_role:1}) [2024-04-24 10:00:51.741584] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [861547][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=618] dump tenant info(tenant={id:1004, tenant_meta:{unit:{tenant_id:1004, unit_id:1003, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1004, name:"c8", resource:{min_cpu:2, max_cpu:8, memory_size:"10.8GB", log_disk_size:"32.4GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:2}}, mode:0, create_timestamp:1713877325237012, is_removed:false}, super_block:{tenant_id:1004, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:false, version:2}, create_status:1}, unit_min_cpu:"2.000000000000000000e+00", unit_max_cpu:"8.000000000000000000e+00", total_worker_cnt:24, min_worker_cnt:10, max_worker_cnt:150, stopped:0, worker_us:20455526256, recv_hp_rpc_cnt:10151, recv_np_rpc_cnt:416515, recv_lp_rpc_cnt:0, recv_mysql_cnt:492540, recv_task_cnt:336, recv_large_req_cnt:0, tt_large_quries:3880, pop_normal_cnt:37286869, workers:10, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=6894 cnt[3]=10 cnt[4]=0 cnt[5]=322815 cnt[6]=114 cnt[7]=196 cnt[8]=0 cnt[9]=0 , group_map:group_id = 2,queue_size = 0,recv_req_cnt = 651407,min_worker_cnt = 2,max_worker_cnt = 150,worker_cnt = 2,token_change = 1713877436490057 group_id = 15,queue_size = 0,recv_req_cnt = 182,min_worker_cnt = 2,max_worker_cnt = 2,worker_cnt = 0,token_change = 1713918462236093 group_id = 5,queue_size = 0,recv_req_cnt = 164331,min_worker_cnt = 2,max_worker_cnt = 150,worker_cnt = 2,token_change = 1713877439477531 group_id = 19,queue_size = 0,recv_req_cnt = 4944,min_worker_cnt = 2,max_worker_cnt = 150,worker_cnt = 1,token_change = 1713923992566938 group_id = 100,queue_size = 0,recv_req_cnt = 394,min_worker_cnt = 8,max_worker_cnt = 150,worker_cnt = 0,token_change = 1713921729471822 group_id = 1,queue_size = 0,recv_req_cnt = 1143231,min_worker_cnt = 8,max_worker_cnt = 8,worker_cnt = 1,token_change = 1713924005029718 , rpc_stat_info: pcode=0x51c:cnt=635 pcode=0x710:cnt=260 pcode=0x1502:cnt=160 pcode=0x650:cnt=102 pcode=0x150b:cnt=80, token_change_ts:1713924002239390, tenant_role:1}) [2024-04-24 10:00:51.744749] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1044) [874489][T1004_FreInfoRe][T1004][Y0-0000000000000000-0-0] [lt=143] success to update min reserved snapshot(reserved_snapshot=1713922246871241000, duration=1800, snapshot_gc_ts_=1713924046871241000) [2024-04-24 10:00:51.745468] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [862402][T1001_TenantMet][T1001][Y0-0000000000000000-0-0] [lt=174] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:769, free_obj_hold_cnt:74, allocator used:3351768, allocator total:3466624}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:72, free_obj_hold_cnt:32, allocator used:6821464, allocator total:6821464}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():154752, full allocator used:1600, full allocator total:220288}, tablets_mem=10508376, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:23, free_obj_hold_cnt:140, allocator used:323392, allocator total:392448}, pending_cnt=0, wait_gc_count=0, tablet count=841) [2024-04-24 10:00:51.752107] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=88] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.753415] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [874491][T1004_ReqMemEvi][T1004][Y0-0000000000000000-0-0] [lt=84] run rewrite rule refresh task(rule_mgr_->tenant_id_=1004) [2024-04-24 10:00:51.755256] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [874491][T1004_ReqMemEvi][T1004][YB420A000003-000616C2B7292B0F-0-0] [lt=84] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924041328327, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924051754221, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:51.755561] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [874491][T1004_ReqMemEvi][T1004][Y0-0000000000000000-0-0] [lt=87] get number(ret=0, service_type_=0, range=1, base_id=1713924051755477000, start_id=1713924051755477000, end_id=1713924051755477001) [2024-04-24 10:00:51.761098] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=104] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.765912] INFO [STORAGE.TRANS] retransmit_downstream_msg_ (ob_two_phase_upstream_committer.cpp:311) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=136] unresponded participant(i=1, *this={this:0x14b055e1a050, ref:5, trans_id:{txid:176911}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051285335], ctx_create_time:1713924048523054}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79627605}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051284482000, v:0}, trans_type:2, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051282008, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877467338312 trans_version={val:1713924051284482000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1 callback_alloc_count=31 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=56 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=56 unsynced_cnt=31 unsubmitted_cnt_=0 cb_statistics:[main=56, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051285620, block_frozen_memtable:null}) [2024-04-24 10:00:51.766274] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=197] handle timeout(ret=0, *this={this:0x14b055e1a050, ref:6, trans_id:{txid:176911}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051285335], ctx_create_time:1713924048523054}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79627605}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051284482000, v:0}, trans_type:2, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051282008, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877467338312 trans_version={val:1713924051284482000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1 callback_alloc_count=31 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=56 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=56 unsynced_cnt=31 unsubmitted_cnt_=0 cb_statistics:[main=56, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051285620, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=200000) [2024-04-24 10:00:51.766432] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=152] trx is waiting log_cb(busy_cbs_.get_size()=2, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:0, v:0}, log_ts:{val:1713924051274688001, v:0}, lsn:{lsn:79627605}, submit_ts:1713924051284482}, this:0x14b055e1c8a8, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():4, range_array_:[{register_no_:1, has_submitted_:true, has_synced_:false, type:1, data_.length():76}, {register_no_:2, has_submitted_:true, has_synced_:false, type:1, data_.length():76}, {register_no_:3, has_submitted_:true, has_synced_:false, type:1, data_.length():67}, {register_no_:4, has_submitted_:true, has_synced_:false, type:6, data_.length():1099}]}, cb_arg_array_:[{log_type:0x4, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924051284482000, v:0}, log_ts:{val:1713924051284482000, v:0}, lsn:{lsn:79629066}, submit_ts:1713924051285620}, this:0x14b055e1caf0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x80, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:51.766753] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:649) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=164] handle timeout(ret=0, *this={this:0x14b055e2e050, ref:5, trans_id:{txid:176911}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924051092564}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88128808}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051278217001, v:0}, trans_type:2, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e30fc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1001}, ctx:0x14b055e2e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=0 trans_version={val:1713924051278217001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051182811, block_frozen_memtable:null}, tx_expired=false, commit_expired=false, delay=200000) [2024-04-24 10:00:51.766946] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:652) [874498][T1004_TransTime][T1004][Y0-0000000000000000-0-0] [lt=186] trx is waiting log_cb(busy_cbs_.get_size()=2, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:0, v:0}, log_ts:{val:1713924051274688001, v:0}, lsn:{lsn:88128808}, submit_ts:1713924051285620}, this:0x14b055e308a8, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1001}, ctx:0x14b055e2e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():2, range_array_:[{register_no_:1, has_submitted_:true, has_synced_:false, type:3, data_.length():159}, {register_no_:2, has_submitted_:true, has_synced_:false, type:1, data_.length():67}]}, cb_arg_array_:[{log_type:0x4, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1713924051278217001, v:0}, log_ts:{val:1713924051278217001, v:0}, lsn:{lsn:88129157}, submit_ts:1713924051286811}, this:0x14b055e30af0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1001}, ctx:0x14b055e2e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x40, arg:null}, {log_type:0x80, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:51.767127] INFO [STORAGE.TRANS] handle_2pc_prepare_request_impl_ (ob_two_phase_downstream_committer.cpp:335) [2322437][T1004_L0_G0][T1004][Y0-0000000000000000-0-0] [lt=197] committer is under logging(ret=0, *this={this:0x14b055e2e050, ref:5, trans_id:{txid:176911}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924051092564}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88128808}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051278217001, v:0}, trans_type:2, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():true, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e30fc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1001}, ctx:0x14b055e2e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=0 trans_version={val:1713924051278217001, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:2, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051182811, block_frozen_memtable:null}) [2024-04-24 10:00:51.767278] INFO [RPC] process (ob_batch_processor.cpp:106) [2322437][T1004_L0_G0][T1004][Y0-0000000000000000-0-0] [lt=142] batch rpc statistics(clog_batch_nodelay_cnt=0, clog_batch_cnt=0, trx_batch_cnt=1, sql_batch_cnt=0) [2024-04-24 10:00:51.768602] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=57][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.780706] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [861520][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=115] new mysql sessid created(conn.sessid_=3221542547, support_ssl=false) [2024-04-24 10:00:51.780848] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [861520][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=77] sm conn init succ(conn.sessid_=3221542547, sess.client_addr_="10.0.0.3:48148") [2024-04-24 10:00:51.781052] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [861520][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=182] accept one succ(*s={this:0x14b11d040030, session_id:3221542547, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:51.781262] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:943) [861520][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=119][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x14b11d040030, session_id:3221542547, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:51.781346] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [861520][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=74] kill and revert session(conn.sessid_=3221542547, proxy_sessid=0, server_id=1, ret=0) [2024-04-24 10:00:51.781463] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:918) [861520][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=87] sock ref clean, do destroy(*s={this:0x14b11d040030, session_id:3221542547, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:51.781549] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:223) [861520][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=74] mark session id unused(conn.sessid_=3221542547) [2024-04-24 10:00:51.781577] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=174] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.781645] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [861520][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=68] connection close(sessid=3221542547, proxy_sessid=0, tenant_id=0, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:0], disconnect_state=0) [2024-04-24 10:00:51.781706] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=118] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.809366] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=69] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.809508] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=127] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.809583] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=67] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.825859] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [862431][T1001_TenantWea][T1001][YB420A000003-000616C2AEECE7EE-0-0] [lt=103] current clock is (clock_=0, max_seq_=0, mem_limit=192099116, cur_mem_hold=130023424, attr_.tenant_id_=1001) [2024-04-24 10:00:51.831600] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0533-0-0] [lt=83][errcode=-4023] tx is 2pc logging(this={this:0x14b055e1a050, ref:6, trans_id:{txid:176911}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051285335], ctx_create_time:1713924048523054}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79627605}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051284482000, v:0}, trans_type:2, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051282008, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877467338312 trans_version={val:1713924051284482000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1 callback_alloc_count=31 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=56 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=56 unsynced_cnt=31 unsubmitted_cnt_=0 cb_statistics:[main=56, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051285620, block_frozen_memtable:null}) [2024-04-24 10:00:51.831839] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0533-0-0] [lt=227][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b055e1a050, ref:6, trans_id:{txid:176911}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051285335], ctx_create_time:1713924048523054}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79627605}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051284482000, v:0}, trans_type:2, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051282008, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():true, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877467338312 trans_version={val:1713924051284482000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1 callback_alloc_count=31 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=56 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=56 unsynced_cnt=31 unsubmitted_cnt_=0 cb_statistics:[main=56, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:30, retain_cause:-1, 2pc_role:0, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051285620, block_frozen_memtable:null}) [2024-04-24 10:00:51.832036] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0533-0-0] [lt=164][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176911}) [2024-04-24 10:00:51.832129] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0533-0-0] [lt=87][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176911}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:39, timestamp:1713924051827089, cluster_id:1713874979}, expire_ts:1713925048510047, commit_start_scn:{val:1713924051278217000, v:0}, parts:[{id:1}, {id:1001}], app_trace_info:""}) [2024-04-24 10:00:51.832209] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0533-0-0] [lt=65][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176911}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:39, timestamp:1713924051827089, cluster_id:1713874979}, expire_ts:1713925048510047, commit_start_scn:{val:1713924051278217000, v:0}, parts:[{id:1}, {id:1001}], app_trace_info:""}) [2024-04-24 10:00:51.833783] WDIAG [STORAGE.TRANS] inner_lock_for_read (ob_tx_data_functor.cpp:290) [2320947][T1004_L0_G0][T1004][YB420A000002-000616C2C30C01F1-0-0] [lt=55][errcode=-6004] lock_for_read need retry(ret=-6004, tx_data={tx_id:{txid:176911}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}, lock_for_read_arg_={mvcc_acc_ctx_:{type:1, abs_lock_timeout:1713924061365787, tx_lock_timeout:-1, snapshot:{version:{val:1713924051424796000, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x14b0efafe490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, data_trans_id_:{txid:176911}, data_sql_sequence_:1713924051282008, read_latest_:false, scn_:{val:4611686018427387903, v:0}}, tx_cc_ctx={state:10, prepare_version:{val:1713924051284482000, v:0}}) [2024-04-24 10:00:51.835335] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=104][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.836078] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=94] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.838994] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [861692][T1_LSArchiveMgr][T1][YB420A000003-000616C2A888F0E7-0-0] [lt=97] gc stale ls task succ [2024-04-24 10:00:51.844816] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=74] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-04-24 10:00:51.852295] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=94] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.859209] INFO [SQL.JO] compute_table_location (ob_join_order.cpp:278) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1EA0-0-0] [lt=65] succeed to calculate base table sharding info(table_id=18446744073709551614, ref_table_id=104, is_global_index=false) [2024-04-24 10:00:51.859364] INFO [SQL.JO] init_est_sel_info_for_access_path (ob_join_order.cpp:12189) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1EA0-0-0] [lt=131] total rowcount, use statistics(table_meta_info_.table_row_count_=10, table_meta_info_.average_row_size_=2.060000000000000000e+02, table_meta_info_.micro_block_count_=1, table_meta_info_.part_size_=2.060000000000000000e+03) [2024-04-24 10:00:51.861245] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=138] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.866112] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [874436][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=157] ====== tenant freeze timer task ====== [2024-04-24 10:00:51.868550] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8EC2751-0-0] [lt=99][errcode=-4023] tx is 2pc logging(this={this:0x14b0fc0a2b50, ref:3, trans_id:{txid:338715}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924081443033, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051470854], ctx_create_time:1713924051456595}{ls_id:{id:1}, session_id:3221795785, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051465459, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924051470594000, v:0}, log_ts:{val:1713924051470594000, v:0}, lsn:{lsn:298527506}, submit_ts:1713924051470594}, this:0x14b0fc0a5ac0, is_inited_:true, trans_id:{txid:338715}, ls_id:{id:1}, ctx:0x14b0fc0a2b50, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338715}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924051470594000, v:0}, start_scn:{val:1713924051470594000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924051470594000, v:0} commit_version={val:1713924051470594000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338715} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3688062677 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924051470594000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051470594, block_frozen_memtable:null}) [2024-04-24 10:00:51.869012] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=109][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.869064] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8EC2751-0-0] [lt=503][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b0fc0a2b50, ref:3, trans_id:{txid:338715}, tenant_id:1003, is_exiting:false, trans_expired_time:1713924081443033, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051470854], ctx_create_time:1713924051456595}{ls_id:{id:1}, session_id:3221795785, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051465459, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924051470594000, v:0}, log_ts:{val:1713924051470594000, v:0}, lsn:{lsn:298527506}, submit_ts:1713924051470594}, this:0x14b0fc0a5ac0, is_inited_:true, trans_id:{txid:338715}, ls_id:{id:1}, ctx:0x14b0fc0a2b50, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0f2804030, tx_data_guard_:{tx_data:{tx_id:{txid:338715}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924051470594000, v:0}, start_scn:{val:1713924051470594000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877383900840 max_table_version=1713877383900840 trans_version={val:1713924051470594000, v:0} commit_version={val:1713924051470594000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:338715} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3688062677 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924051470594000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051470594, block_frozen_memtable:null}) [2024-04-24 10:00:51.869354] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8EC2751-0-0] [lt=246][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:338715}) [2024-04-24 10:00:51.869432] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8EC2751-0-0] [lt=72][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338715}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924051863553, cluster_id:1713874979}, expire_ts:1713924081466140, commit_start_scn:{val:1713924051458925000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:51.870991] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2248569][T1003_L0_G0][T1003][YB420A000002-000616C2B8EC2751-0-0] [lt=1547][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1003, tx_id:{txid:338715}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924051863553, cluster_id:1713874979}, expire_ts:1713924081466140, commit_start_scn:{val:1713924051458925000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:51.871118] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [861689][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=103] ====== check clog disk timer task ====== [2024-04-24 10:00:51.871215] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [861689][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=85] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=235) [2024-04-24 10:00:51.871360] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [861689][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=102] cannot_recycle_log_size statistics(cannot_recycle_log_size=112644894, threshold=644245094, need_update_checkpoint_scn=false) [2024-04-24 10:00:51.881742] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=115] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.881842] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=92] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.886563] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [874531][T1004_TabletShe][T1004][Y0-0000000000000000-0-0] [lt=101] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-04-24 10:00:51.886714] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [874531][T1004_TabletShe][T1004][Y0-0000000000000000-0-0] [lt=75] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=3, times=9344) [2024-04-24 10:00:51.897429] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [861461][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=69] skip inspect bad block(last_check_time=1713875167993818, last_macro_idx=-1) [2024-04-24 10:00:51.904761] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [862460][T1001_L0_G0][T1001][YB420A000002-000616C2B27C260C-0-0] [lt=122][errcode=-4023] tx is 2pc logging(this={this:0x14b15a15d590, ref:3, trans_id:{txid:391373}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924081449792, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051475560], ctx_create_time:1713924051461248}{ls_id:{id:1}, session_id:3221795795, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051472069, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924051474521000, v:0}, log_ts:{val:1713924051474521000, v:0}, lsn:{lsn:314135992}, submit_ts:1713924051474521}, this:0x14b15a160500, is_inited_:true, trans_id:{txid:391373}, ls_id:{id:1}, ctx:0x14b15a15d590, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:391373}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924051474521000, v:0}, start_scn:{val:1713924051474521000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875442714112 max_table_version=1713875442714112 trans_version={val:1713924051474521000, v:0} commit_version={val:1713924051474521000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:391373} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=2895691470 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924051474521000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051475837, block_frozen_memtable:null}) [2024-04-24 10:00:51.905030] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [862460][T1001_L0_G0][T1001][YB420A000002-000616C2B27C260C-0-0] [lt=249][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b15a15d590, ref:3, trans_id:{txid:391373}, tenant_id:1001, is_exiting:false, trans_expired_time:1713924081449792, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051475560], ctx_create_time:1713924051461248}{ls_id:{id:1}, session_id:3221795795, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924051472069, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924051474521000, v:0}, log_ts:{val:1713924051474521000, v:0}, lsn:{lsn:314135992}, submit_ts:1713924051474521}, this:0x14b15a160500, is_inited_:true, trans_id:{txid:391373}, ls_id:{id:1}, ctx:0x14b15a15d590, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b14b804030, tx_data_guard_:{tx_data:{tx_id:{txid:391373}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924051474521000, v:0}, start_scn:{val:1713924051474521000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713875442714112 max_table_version=1713875442714112 trans_version={val:1713924051474521000, v:0} commit_version={val:1713924051474521000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:391373} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=2895691470 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924051474521000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051475837, block_frozen_memtable:null}) [2024-04-24 10:00:51.905209] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [862460][T1001_L0_G0][T1001][YB420A000002-000616C2B27C260C-0-0] [lt=142][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:391373}) [2024-04-24 10:00:51.905598] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [862460][T1001_L0_G0][T1001][YB420A000002-000616C2B27C260C-0-0] [lt=379][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1001, tx_id:{txid:391373}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924051900087, cluster_id:1713874979}, expire_ts:1713924081470822, commit_start_scn:{val:1713924051462734001, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:51.905715] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [862460][T1001_L0_G0][T1001][YB420A000002-000616C2B27C260C-0-0] [lt=95][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1001, tx_id:{txid:391373}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924051900087, cluster_id:1713874979}, expire_ts:1713924081470822, commit_start_scn:{val:1713924051462734001, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:51.906018] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1EA0-0-0] [lt=82][errcode=0] set logical del time(cache_obj->get_logical_del_time()=576364480138, cache_obj->added_lc()=false, cache_obj->get_object_id()=39600, cache_obj->get_tenant_id()=1, lbt()="0x121f36ac 0xc327dd4 0x4e3fdbe 0x4e31e28 0x4e29b67 0x4e21201 0x4e1eb17 0x4e0e175 0xa7ac354 0x12d1fdbc 0x14b1c1d132de 0x14b1c1a44e83") [2024-04-24 10:00:51.909554] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=102] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.909711] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=137] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.909817] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=98] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.925777] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=77] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-24 10:00:51.926135] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=130] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=276, get_gts_with_stc_cnt=25, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=20, try_wait_gts_elapse_cnt=0) [2024-04-24 10:00:51.926533] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [861494][TsMgr][T1004][Y0-0000000000000000-0-0] [lt=120] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=11806) [2024-04-24 10:00:51.926695] INFO [LIB] stat (utility.h:1197) [861494][TsMgr][T1004][Y0-0000000000000000-0-0] [lt=117] [PALF STAT APPEND COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x14b0efafce50, extra_info="id:1") [2024-04-24 10:00:51.926801] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:157) [861494][TsMgr][T1004][Y0-0000000000000000-0-0] [lt=93] submit log success(service_type_=0, last_id=1713924068572929000, limited_id=1713924068572929000) [2024-04-24 10:00:51.926940] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [861494][TsMgr][T1004][Y0-0000000000000000-0-0] [lt=128] submit log with lock success(service_type_=0, last_id=1713924068572929000, limited_id=1713924068572929000) [2024-04-24 10:00:51.927110] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=125] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8903270400, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:51.930733] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=141] new mysql sessid created(conn.sessid_=3221542653, support_ssl=false) [2024-04-24 10:00:51.930971] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=115] sm conn init succ(conn.sessid_=3221542653, sess.client_addr_="10.0.0.3:48650") [2024-04-24 10:00:51.931062] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=72] accept one succ(*s={this:0x14b0c23fa030, session_id:3221542653, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:51.931338] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [861518][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=76] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-04-24 10:00:51.931412] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:280) [861518][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=65] mysql login queue(mysql_queue->queue_.size()=0) [2024-04-24 10:00:51.932415] WDIAG [COMMON] get (ob_link_hashmap.h:313) [861676][T1_MysqlQueueTh][T1][Y0-000616C2A9892E66-0-0] [lt=73][errcode=0] uref leak check(node->uref_=1073788696, lbt()="0x121f36ac 0x5547c4b 0xac68123 0xac676b8 0xac3f33a 0xa3599e4 0xa3534fa 0x4e21201 0x13857852 0x13858aaa 0x13859029 0x12d23b01 0x12d1fdbc 0x14b1c1d132de 0x14b1c1a44e83") [2024-04-24 10:00:51.932805] INFO [SERVER] process (obmp_connect.cpp:475) [861676][T1_MysqlQueueTh][T1][Y0-000616C2A9892E66-0-0] [lt=62] MySQL LOGIN(direct_client_ip="10.0.0.3", client_ip=10.0.0.3, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221542653, proxy_sessid=720575953453123048, sess_create_time=0, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=916303, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-04-24 10:00:51.933861] INFO [SERVER] process (obmp_init_db.cpp:53) [861896][T1_L0_G0][T1][YB420A000003-000616C2AD78FD02-0-0] [lt=67] init db(db_name=oceanbase) [2024-04-24 10:00:51.935511] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=90][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.936296] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=128] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.936389] WDIAG [SQL] handle_text_query (ob_sql.cpp:2663) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1EA3-0-0] [lt=69][errcode=-5727] fail to handle after get plan(ret=-5727) [2024-04-24 10:00:51.936572] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:549) [861898][T1_L0_G0][T1][YB420A000003-000616C2ABEB1EA3-0-0] [lt=95] dump txn free route audit_record(value=5, session->get_sessid()=3221542653, session->get_proxy_sessid()=720575953453123048) [2024-04-24 10:00:51.938116] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:943) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=91][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x14b0c23fa030, session_id:3221542653, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:1713924051934955, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:695, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:51.938240] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=110] kill and revert session(conn.sessid_=3221542653, proxy_sessid=720575953453123048, server_id=1, ret=0) [2024-04-24 10:00:51.938334] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:918) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=81] sock ref clean, do destroy(*s={this:0x14b0c23fa030, session_id:3221542653, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:1713924051934955, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:695, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:51.938516] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:159) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=94] unlock session of tenant(conn.sessid_=3221542653, proxy_sessid=720575953453123048, conn.tenant_id_=1) [2024-04-24 10:00:51.938715] WDIAG [COMMON] get (ob_link_hashmap.h:313) [861898][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=120][errcode=0] uref leak check(node->uref_=1073788696, lbt()="0x121f36ac 0x5547c4b 0xac68123 0xac69880 0x521a363 0x51fb0b7 0xa36ef16 0x4e1feed 0x4e0e175 0xa7ac354 0x12d1fdbc 0x14b1c1d132de 0x14b1c1a44e83") [2024-04-24 10:00:51.938646] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=112] connection close(sessid=3221542653, proxy_sessid=720575953453123048, tenant_id=1, server_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB420A000003-000616C2ABEB1EA3-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:4, cur_pkt_pos_:6, pkt_rec[0]:{send:obp_mysql_header_{len_:6368, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:5963473, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:550, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:5963474, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:412, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:261, pkt_seq:0, req_id:5963475, flag_.flags:62, ext_flags:0}, obp_mysql_header_{len_:257, rec_:261, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:222, com_seq:1}, obp20_header_:{payload_len:194, pkt_seq:1, req_id:5963475, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:43, seq_:2}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:222, com_seq:1}, obp20_header_:{payload_len:194, pkt_seq:1, req_id:5963475, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:151, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2024-04-24 10:00:51.938936] INFO [SERVER] run (obmp_disconnect.cpp:82) [861898][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=99] free session successfully(sessid=3221542653) [2024-04-24 10:00:51.939091] INFO [SERVER] run (obmp_disconnect.cpp:89) [861898][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=138] mark session id unused(sessid=3221542653) [2024-04-24 10:00:51.943324] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [862266][T1002_LSArchive][T1002][YB420A000003-000616C2AE58F0E7-0-0] [lt=65] gc stale ls task succ [2024-04-24 10:00:51.952397] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=89] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.957447] INFO eloop_run (eloop.c:143) [861404][pnio2][T0][Y0-0000000000000000-0-0] [lt=147] PNIO [ratelimit] time: 1713924051957441, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007456, add_bytes: 0 [2024-04-24 10:00:51.961500] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=68] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:51.969187] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=92][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:51.974829] INFO [LIB] stat (utility.h:1197) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=101] [PALF STAT WRITE LOG COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=386782, this=0x14b14c9bd3b0, extra_info="palf_id:1") [2024-04-24 10:00:51.975052] INFO [LIB] stat (utility.h:1197) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=157] [PALF STAT WRITE LOG COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=256828, this=0x14b0efb333b0, extra_info="palf_id:1") [2024-04-24 10:00:51.975095] INFO [LIB] stat (utility.h:1197) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=96] [PALF STAT WRITE LOG COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=426619, this=0x14b13adcd3b0, extra_info="palf_id:1002") [2024-04-24 10:00:51.975059] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=223][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:314134855}], scn_array=[{val:1713924051114377000, v:0}], curr_size=1081, accum_size=2231, time_cost=530449) [2024-04-24 10:00:51.975122] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [874495][T1004_OB_SLOG][T1004][Y0-0000000000000000-0-0] [lt=162] Slow write(duration=712106, write_len=4096) [2024-04-24 10:00:51.975172] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=104][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:79627427}], scn_array=[{val:1713924051274688000, v:0}], curr_size=122, accum_size=777, time_cost=694125) [2024-04-24 10:00:51.975218] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=113][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:165105848}], scn_array=[{val:1713924050962740001, v:0}], curr_size=122, accum_size=1695, time_cost=530693) [2024-04-24 10:00:51.975218] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=149] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=2231) [2024-04-24 10:00:51.975263] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [874495][T1004_OB_SLOG][T1004][Y0-0000000000000000-0-0] [lt=96] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=93235, offset=21596911}, end_cursor:ObLogCursor{file_id=1, log_id=93236, offset=21597152}, is_inited:true, is_local:false, buf_size:8192, buf:0x14b19ef48050, len:1297, log_data_len:241, seq:93235, flush_finish:false, flush_ret:0}) [2024-04-24 10:00:51.975306] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=127] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=777) [2024-04-24 10:00:51.975339] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=112] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=1695) [2024-04-24 10:00:51.975424] INFO [LIB] stat (utility.h:1214) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=130] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=181, this=0x14b157ff6530, extra_info="") [2024-04-24 10:00:51.975444] INFO [LIB] stat (utility.h:1214) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=86] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=503, this=0x14b176ffc530, extra_info="") [2024-04-24 10:00:51.975474] INFO [LIB] stat (utility.h:1214) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=85] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=91753, this=0x14b1679f73b0, extra_info="") [2024-04-24 10:00:51.975517] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=110] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=3017) [2024-04-24 10:00:51.975547] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=112] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=1692) [2024-04-24 10:00:51.975603] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:172) [874504][T1004_CdcSrv][T1004][Y0-0000000000000000-0-0] [lt=107] query dest_info_ succ(dest_info=[], archive_dest_changed=false) [2024-04-24 10:00:51.975771] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:632) [874504][T1004_CdcSrv][T1004][Y0-0000000000000000-0-0] [lt=139] finish to resize log external storage handler(current_ts=1713924051975756, tenant_max_cpu=8, valid_ls_count=0, other_ls_count=0, new_concurrency=0) [2024-04-24 10:00:51.975818] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=95] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=122, this=0x14b0efb29470, extra_info="") [2024-04-24 10:00:51.975784] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=112] [PALF STAT FLUSH CB COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=133, this=0x14b13adcd5e0, extra_info="palf_id:1002") [2024-04-24 10:00:51.976048] INFO [LIB] stat (utility.h:1197) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=188] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=131, this=0x14b14c9b3470, extra_info="") [2024-04-24 10:00:51.976061] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=184] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=3, avg_log_batch_cnt=2, total_group_log_size=11436, avg_group_log_size=3812) [2024-04-24 10:00:51.976206] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=93] [PALF STAT FLUSH CB COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=343, this=0x14b0efb335e0, extra_info="palf_id:1") [2024-04-24 10:00:51.976206] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=97] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=3, avg_log_batch_cnt=2, total_group_log_size=3236, avg_group_log_size=1078) [2024-04-24 10:00:51.976328] INFO [LIB] stat (utility.h:1197) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=81] [PALF STAT FLUSH CB COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=400, this=0x14b14c9bd5e0, extra_info="palf_id:1") [2024-04-24 10:00:51.976628] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F7D-0-0] [lt=89] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=4093) [2024-04-24 10:00:51.976933] INFO [PALF] inner_write_once_ (log_block_handler.cpp:411) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=112] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=30111446, this={dio_aligned_buf:{buf_write_offset:2778, buf_padding_size:1318, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:5876129, truncate_used_ts:6990448}, log_block_size:67108864, dir_fd:199, io_fd:147}, aligned_buf_len=4096, aligned_buf="ì¨ÝŸå²Ø…¦ª", aligned_block_offset=30109696, buf_len=1028, total_write_size_=43319, total_write_size_after_dio_=307200, ob_pwrite_used_ts=6724067, count=65, each_pwrite_cost=103447) [2024-04-24 10:00:51.977142] INFO [LIB] stat (utility.h:1197) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=166] [PALF STAT WRITE LOG COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=216205, this=0x14b0f0aab3b0, extra_info="palf_id:1") [2024-04-24 10:00:51.977243] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=92][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:298526422}], scn_array=[{val:1713924051219710000, v:0}], curr_size=1028, accum_size=4148, time_cost=531195) [2024-04-24 10:00:51.977356] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=103] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=4148) [2024-04-24 10:00:51.977468] INFO [LIB] stat (utility.h:1214) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=71] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=553, this=0x14b1421a2530, extra_info="") [2024-04-24 10:00:51.977733] INFO [LIB] stat (utility.h:1197) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F7D-0-0] [lt=1018] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=27, this=0x14b0f0aa16a0, extra_info="palf_id:1") [2024-04-24 10:00:51.977785] INFO [LIB] stat (utility.h:1197) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=91] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=113, this=0x14b0f0aa1470, extra_info="") [2024-04-24 10:00:51.977851] INFO [LIB] stat (utility.h:1197) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F7D-0-0] [lt=98] [PALF STAT LOG LIFE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=876729, this=0x14b0f0aa18d0, extra_info="palf_id:1") [2024-04-24 10:00:51.977971] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=77] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=5, avg_log_batch_cnt=1, total_group_log_size=4405, avg_group_log_size=881) [2024-04-24 10:00:51.978009] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F7D-0-0] [lt=148] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=4, avg_log_gen_to_freeze_time=427928, avg_log_gen_to_submit_time=428123, avg_log_submit_to_flush_time=137941, avg_log_submit_to_slide_time=448606) [2024-04-24 10:00:51.978032] INFO [LIB] stat (utility.h:1197) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=77] [APPLY STAT CB APPEND COST TIME](cur_stat_count=10, stat_interval=5000000, avg (count/cost)=148, this=0x14b114df4280, extra_info="") [2024-04-24 10:00:51.978264] INFO [LIB] stat (utility.h:1197) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=140] [PALF STAT FLUSH CB COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=317, this=0x14b0f0aab5e0, extra_info="palf_id:1") [2024-04-24 10:00:51.978264] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F7D-0-0] [lt=231][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=482559, log_task={header:{begin_lsn:{lsn:298526301}, end_lsn:{lsn:298526422}, log_id:482559, min_scn:{val:1713924050970319001, v:0}, max_scn:{val:1713924050970319001, v:0}, data_len:65, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:298524396}, data_checksum:1773765733, accum_checksum:2189551668, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924051161424, freeze_ts:1713924051161431, submit_ts:1713924051161539, flushed_ts:1713924051443714, gen_to_freeze cost time:7, gen_to_submit cost time:115, submit_to_flush cost time:282175}, fs_cb_begin_ts=1713924051977681, log_life_time=816257) [2024-04-24 10:00:51.978295] INFO [LIB] stat (utility.h:1197) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=241] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=10, stat_interval=5000000, avg (count/cost)=595782, this=0x14b114df44b0, extra_info="") [2024-04-24 10:00:51.978309] INFO [PALF] inner_write_once_ (log_block_handler.cpp:411) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=166] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=45544222, this={dio_aligned_buf:{buf_write_offset:1882, buf_padding_size:2214, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:4332756, truncate_used_ts:5271173}, log_block_size:67108864, dir_fd:145, io_fd:148}, aligned_buf_len=4096, aligned_buf="", aligned_block_offset=45543424, buf_len=1084, total_write_size_=74918, total_write_size_after_dio_=393216, ob_pwrite_used_ts=6601455, count=78, each_pwrite_cost=84634) [2024-04-24 10:00:51.978315] INFO [PALF] inner_write_once_ (log_block_handler.cpp:411) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=144] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=40365165, this={dio_aligned_buf:{buf_write_offset:4018, buf_padding_size:78, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:3611049, truncate_used_ts:4431852}, log_block_size:67108864, dir_fd:209, io_fd:213}, aligned_buf_len=4096, aligned_buf="˜€€€", aligned_block_offset=40361984, buf_len=837, total_write_size_=13677, total_write_size_after_dio_=172032, ob_pwrite_used_ts=5959376, count=39, each_pwrite_cost=152804) [2024-04-24 10:00:51.978524] INFO [LIB] stat (utility.h:1197) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=167] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=10, stat_interval=5000000, avg (count/cost)=0, this=0x14b114df46e0, extra_info="") [2024-04-24 10:00:51.978511] INFO [LIB] stat (utility.h:1197) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=69] [PALF STAT WRITE LOG COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=303843, this=0x14b1470b53b0, extra_info="palf_id:1") [2024-04-24 10:00:51.978561] INFO [LIB] stat (utility.h:1197) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=223] [PALF STAT WRITE LOG COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=424331, this=0x14b186b513b0, extra_info="palf_id:1") [2024-04-24 10:00:51.978561] INFO [LIB] stat (utility.h:1197) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=217] [PALF STAT WRITE LOG COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=286594, this=0x14b0c0f013b0, extra_info="palf_id:1002") [2024-04-24 10:00:51.978720] INFO [LIB] stat (utility.h:1197) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=124] [APPLY STAT CB EXECUTE TIME](cur_stat_count=10, stat_interval=5000000, avg (count/cost)=-374, this=0x14b114df4910, extra_info="") [2024-04-24 10:00:51.978720] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=182][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67160209}], scn_array=[{val:1713924051045053001, v:0}], curr_size=122, accum_size=870, time_cost=531867) [2024-04-24 10:00:51.978777] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=200][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179749662}], scn_array=[{val:1713924051459674000, v:0}], curr_size=1084, accum_size=2094, time_cost=513046) [2024-04-24 10:00:51.978785] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=216][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:40361069}], scn_array=[{val:1713924051138765000, v:0}], curr_size=837, accum_size=3173, time_cost=714730) [2024-04-24 10:00:51.978850] INFO [LIB] stat (utility.h:1197) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=110] [APPLY STAT CB TOTAL TIME](cur_stat_count=10, stat_interval=5000000, avg (count/cost)=595556, this=0x14b114df4b40, extra_info="") [2024-04-24 10:00:51.978857] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=110] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=870) [2024-04-24 10:00:51.978863] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=80] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=2094) [2024-04-24 10:00:51.978945] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=154] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=3173) [2024-04-24 10:00:51.979043] INFO [LIB] stat (utility.h:1214) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=140] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=225, this=0x14b1945fe530, extra_info="") [2024-04-24 10:00:51.979061] INFO [LIB] stat (utility.h:1214) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=136] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=594, this=0x14b1679f6530, extra_info="") [2024-04-24 10:00:51.979061] INFO [LIB] stat (utility.h:1214) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=71] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=564, this=0x14b176ffd3b0, extra_info="") [2024-04-24 10:00:51.979201] INFO [LIB] stat (utility.h:1197) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=111] [PALF STAT LOG IO WORKER QUEUE SIZE](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=1, this=0x14b176ffd068, extra_info="log_io_worker_num:4, cb_thread_pool_tg_id:591") [2024-04-24 10:00:51.979270] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=81] [PALF STAT FLUSH CB COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=166, this=0x14b0c0f015e0, extra_info="palf_id:1002") [2024-04-24 10:00:51.979504] INFO [LIB] stat (utility.h:1197) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=168] [PALF STAT WRITE LOG COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=452260, this=0x14b1336f53b0, extra_info="palf_id:1001") [2024-04-24 10:00:51.979633] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=115][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:1002197139}], scn_array=[{val:1713924051423315000, v:0}], curr_size=122, accum_size=244, time_cost=470185) [2024-04-24 10:00:51.979707] INFO [PALF] inner_write_once_ (log_block_handler.cpp:411) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=187] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=21027065, this={dio_aligned_buf:{buf_write_offset:3872, buf_padding_size:224, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:4182164, truncate_used_ts:4991395}, log_block_size:67108864, dir_fd:201, io_fd:134}, aligned_buf_len=4096, aligned_buf="F_BYTE_USED","charsetMapping":{"utf8mb4":"utf8mb4","ascii":"latin1"},"collationMapping":{"ascii_general_ci":"latin1_swedish_ci","utf8mb4_general_ci":"utf8mb4_general_ci","utf8mb4_0900_ai_ci":"utf8mb4_general_ci"},"enableIndexBehind":false,"maxConnectorSize":10,"migrationScenes":"ONE_WAY","partition":0,"tableCategory":"ALL","taskIdentity":"np_5of4u13d37a8","taskType":"MIGRATION","user":"admin"}", aligned_block_offset=21024768, buf_len=1575, total_write_size_=46400, total_write_size_after_dio_=311296, ob_pwrite_used_ts=6079973, count=65, each_pwrite_cost=93538) [2024-04-24 10:00:51.979754] INFO [LIB] stat (utility.h:1197) [861568][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=74] [PALF STAT FLUSH CB COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=152, this=0x14b186b515e0, extra_info="palf_id:1") [2024-04-24 10:00:51.979811] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=164] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=244) [2024-04-24 10:00:51.979946] INFO [LIB] stat (utility.h:1197) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=142] [PALF STAT WRITE LOG COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=195275, this=0x14b0c0e353b0, extra_info="palf_id:1001") [2024-04-24 10:00:51.980026] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=69][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:88127737}], scn_array=[{val:1713924051274688000, v:0}], curr_size=1575, accum_size=5580, time_cost=530032) [2024-04-24 10:00:51.980038] INFO [LIB] stat (utility.h:1214) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=187] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=202, this=0x14b1679f6c70, extra_info="") [2024-04-24 10:00:51.980050] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=229] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=870) [2024-04-24 10:00:51.980166] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [874422][T1004_TeRec][T1003][YB420A000003-000616C2B5596E96-0-0] [lt=74] current clock is (clock_=0, max_seq_=0, mem_limit=264912214, cur_mem_hold=121634816, attr_.tenant_id_=1003) [2024-04-24 10:00:51.980166] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=126] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=5580) [2024-04-24 10:00:51.980308] INFO [LIB] stat (utility.h:1214) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=83] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=1581, this=0x14b176ffcc70, extra_info="") [2024-04-24 10:00:51.980332] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=141] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=92, this=0x14b1470ab470, extra_info="") [2024-04-24 10:00:51.980341] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=141] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="10.0.0.3:2882", committed size=5829) [2024-04-24 10:00:51.980430] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=4, avg_log_batch_cnt=1, total_group_log_size=1508, avg_group_log_size=377) [2024-04-24 10:00:51.980556] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=81] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=85, this=0x14b0c0e2b470, extra_info="") [2024-04-24 10:00:51.980582] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=76] [PALF STAT FLUSH CB COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=284, this=0x14b1470b55e0, extra_info="palf_id:1") [2024-04-24 10:00:51.980712] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=117] [PALF STAT GROUP LOG INFO](palf_id=1001, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=4, avg_log_batch_cnt=1, total_group_log_size=6025, avg_group_log_size=1506) [2024-04-24 10:00:51.980812] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=130] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="10.0.0.3:2882", committed size=244) [2024-04-24 10:00:51.980982] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=140] [PALF STAT FLUSH CB COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=311, this=0x14b0c0e355e0, extra_info="palf_id:1001") [2024-04-24 10:00:51.981042] INFO [LIB] stat (utility.h:1197) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=179] [PALF STAT FLUSH CB COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=253, this=0x14b1336f55e0, extra_info="palf_id:1001") [2024-04-24 10:00:51.981798] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [874422][T1004_TeRec][T1003][YB420A000003-000616C2B5596E96-0-0] [lt=77] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=4405) [2024-04-24 10:00:51.982105] INFO [LIB] stat (utility.h:1197) [874422][T1004_TeRec][T1003][YB420A000003-000616C2B5596E96-0-0] [lt=208] [PALF STAT APPEND COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x14b0f0a74e50, extra_info="id:1") [2024-04-24 10:00:51.982161] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=144] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:51.982337] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=125] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:51.984801] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD2A2-0-0] [lt=170] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=25, this=0x14b0efb296a0, extra_info="palf_id:1") [2024-04-24 10:00:51.984998] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD2A2-0-0] [lt=183] [PALF STAT LOG LIFE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=817639, this=0x14b0efb298d0, extra_info="palf_id:1") [2024-04-24 10:00:51.985073] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD2A2-0-0] [lt=63] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=4, avg_log_gen_to_freeze_time=7864, avg_log_gen_to_submit_time=8033, avg_log_submit_to_flush_time=602750, avg_log_submit_to_slide_time=809606) [2024-04-24 10:00:51.985080] INFO [LIB] stat (utility.h:1197) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB904-0-0] [lt=115] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=24, this=0x14b14c9b36a0, extra_info="palf_id:1") [2024-04-24 10:00:51.985168] INFO [LIB] stat (utility.h:1197) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB904-0-0] [lt=73] [PALF STAT LOG LIFE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=1150846, this=0x14b14c9b38d0, extra_info="palf_id:1") [2024-04-24 10:00:51.985153] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD2A2-0-0] [lt=70][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=304865, log_task={header:{begin_lsn:{lsn:79627427}, end_lsn:{lsn:79627549}, log_id:304865, min_scn:{val:1713924051274688000, v:0}, max_scn:{val:1713924051274688000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:79627427}, data_checksum:667309872, accum_checksum:1114459452, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924051280568, freeze_ts:1713924051280581, submit_ts:1713924051280735, flushed_ts:1713924051975496, gen_to_freeze cost time:13, gen_to_submit cost time:167, submit_to_flush cost time:694761}, fs_cb_begin_ts=1713924051984745, log_life_time=704177) [2024-04-24 10:00:51.985209] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=170][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051114377000, v:0}, log_ts:{val:1713924051114377000, v:0}, lsn:{lsn:314134911}, submit_ts:1713924051114377}, this:0x14b15a14f340, is_inited_:true, trans_id:{txid:568576}, ls_id:{id:1}, ctx:0x14b15a14c3d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=870810, ctx_lock_wait_time=13) [2024-04-24 10:00:51.985236] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB904-0-0] [lt=61] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=3, avg_log_gen_to_freeze_time=704676, avg_log_gen_to_submit_time=704868, avg_log_submit_to_flush_time=387431, avg_log_submit_to_slide_time=445977) [2024-04-24 10:00:51.985325] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB904-0-0] [lt=77][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=505467, log_task={header:{begin_lsn:{lsn:314134855}, end_lsn:{lsn:314135936}, log_id:505467, min_scn:{val:1713924051114377000, v:0}, max_scn:{val:1713924051114377001, v:0}, data_len:1025, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:314134855}, data_checksum:3653983526, accum_checksum:1579578611, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924051115844, freeze_ts:1713924051443947, submit_ts:1713924051444093, flushed_ts:1713924051975517, gen_to_freeze cost time:328103, gen_to_submit cost time:328249, submit_to_flush cost time:531424}, fs_cb_begin_ts=1713924051985033, log_life_time=869189) [2024-04-24 10:00:51.988272] INFO [SQL.PC] check_can_do_insert_opt (ob_plan_cache.cpp:996) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2C6-0-0] [lt=75] can not do batch insert opt(ret=0, can_do_batch=false, upd_params_count=0, ins_params_count=0, batch_count=0, pc_ctx.raw_sql_=insert into heart_beat (id ,task_type ,task_name ,task_id ,host_ip ,version, gmt_created ,gmt_modified ) values (3000241,'supervisor','supervisor', null,'10.0.0.3',null,now(),now()) ON DUPLICATE KEY UPDATE version = VALUES(version), gmt_modified= now()) [2024-04-24 10:00:51.988552] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2C6-0-0] [lt=168] is logging [2024-04-24 10:00:51.988734] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2C6-0-0] [lt=165] submit log with lock(ret=-4023, service_type_=0, last_id=1713924068572929000, limited_id=1713924068572929000) [2024-04-24 10:00:51.990964] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:860) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2C6-0-0] [lt=103] memtable handle rollback to successfuly(from_seq_no=1713924051985207, to_seq_no=1713924051985202, *this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713880252096976 max_table_version=1713880252096976 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:327211} ls_id=1001 callback_alloc_count=3 callback_free_count=3 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=7, slave=0, merge=0, tx_end=0, rollback_to=7, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}) [2024-04-24 10:00:51.991668] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C51-0-0] [lt=125] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=27, this=0x14b1470ab6a0, extra_info="palf_id:1") [2024-04-24 10:00:51.991817] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C51-0-0] [lt=132] [PALF STAT LOG LIFE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=604692, this=0x14b1470ab8d0, extra_info="palf_id:1") [2024-04-24 10:00:51.991939] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C51-0-0] [lt=115] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=4, avg_log_gen_to_freeze_time=259623, avg_log_gen_to_submit_time=259780, avg_log_submit_to_flush_time=305349, avg_log_submit_to_slide_time=344912) [2024-04-24 10:00:51.992037] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C51-0-0] [lt=91][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=259322, log_task={header:{begin_lsn:{lsn:67160209}, end_lsn:{lsn:67160331}, log_id:259322, min_scn:{val:1713924051045053001, v:0}, max_scn:{val:1713924051045053001, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:67159711}, data_checksum:667309872, accum_checksum:2817567938, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924051408557, freeze_ts:1713924051446385, submit_ts:1713924051446469, flushed_ts:1713924051980026, gen_to_freeze cost time:37828, gen_to_submit cost time:37912, submit_to_flush cost time:533557}, fs_cb_begin_ts=1713924051991615, log_life_time=583058) [2024-04-24 10:00:51.992305] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2C6-0-0] [lt=105] is logging [2024-04-24 10:00:51.992434] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2C6-0-0] [lt=104] submit log with lock(ret=-4023, service_type_=0, last_id=1713924068572929000, limited_id=1713924068572929000) [2024-04-24 10:00:51.992494] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=143][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051219710000, v:0}, log_ts:{val:1713924051219710000, v:0}, lsn:{lsn:298526478}, submit_ts:1713924051219710}, this:0x14b0fbee1fc0, is_inited_:true, trans_id:{txid:548737}, ls_id:{id:1}, ctx:0x14b0fbedf050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=772769, ctx_lock_wait_time=8) [2024-04-24 10:00:51.992590] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2C6-0-0] [lt=76] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, append size=6182) [2024-04-24 10:00:51.992710] INFO [LIB] stat (utility.h:1197) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2C6-0-0] [lt=74] [PALF STAT APPEND COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x14b0c0f0ae50, extra_info="id:1001") [2024-04-24 10:00:51.992775] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=111][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051418462000, v:0}, log_ts:{val:1713924051418462000, v:0}, lsn:{lsn:298526964}, submit_ts:1713924051418462}, this:0x14b0fc097fc0, is_inited_:true, trans_id:{txid:548738}, ls_id:{id:1}, ctx:0x14b0fc095050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=574296, ctx_lock_wait_time=9) [2024-04-24 10:00:51.993085] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [874526][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=154] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1004, new_min_version={val:1713924048598120000, v:0}, new_max_version={val:1713924048598120000, v:0}, last_min_version={val:1713924048463546001, v:0}, last_max_version={val:1713924048463546001, v:0}, total_time_us=775090, delta_us=3394957) [2024-04-24 10:00:51.993211] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [874526][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=107] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1004, current_version={val:1713924048598120000, v:0}, min_version={val:1713924048598120000, v:0}, max_version={val:1713924048598120000, v:0}, in_service=true, leader_epoch=5) [2024-04-24 10:00:51.993370] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C52-0-0] [lt=147] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=52, this=0x14b1336eb6a0, extra_info="palf_id:1001") [2024-04-24 10:00:51.993486] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C52-0-0] [lt=99] [PALF STAT LOG LIFE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=459829, this=0x14b1336eb8d0, extra_info="palf_id:1001") [2024-04-24 10:00:51.993591] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C52-0-0] [lt=99] [PALF STAT LOG TASK TIME](palf_id=1001, self="10.0.0.3:2882", total_slide_log_cnt=2, avg_log_gen_to_freeze_time=19, avg_log_gen_to_submit_time=248, avg_log_submit_to_flush_time=453314, avg_log_submit_to_slide_time=459581) [2024-04-24 10:00:51.993746] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C52-0-0] [lt=143][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=673426, log_task={header:{begin_lsn:{lsn:1002197139}, end_lsn:{lsn:1002197261}, log_id:673426, min_scn:{val:1713924051423315000, v:0}, max_scn:{val:1713924051423315000, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1002197139}, data_checksum:2222604892, accum_checksum:3288112856, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924051508902, freeze_ts:1713924051508915, submit_ts:1713924051509109, flushed_ts:1713924051980789, gen_to_freeze cost time:13, gen_to_submit cost time:207, submit_to_flush cost time:471680}, fs_cb_begin_ts=1713924051993283, log_life_time=484381) [2024-04-24 10:00:51.993854] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD141F-0-0] [lt=104][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=364247, log_task={header:{begin_lsn:{lsn:179749662}, end_lsn:{lsn:179750746}, log_id:364247, min_scn:{val:1713924051459674000, v:0}, max_scn:{val:1713924051459674000, v:0}, data_len:1028, proposal_id:5, prev_lsn:{lsn:179748652}, prev_proposal_id:5, committed_end_lsn:{lsn:179749662}, data_checksum:89761966, accum_checksum:4047803643, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924051465298, freeze_ts:1713924051465305, submit_ts:1713924051465359, flushed_ts:1713924051979537, gen_to_freeze cost time:7, gen_to_submit cost time:61, submit_to_flush cost time:514178}, fs_cb_begin_ts=1713924051993793, log_life_time=528495) [2024-04-24 10:00:51.998287] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [874526][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=159] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1004, server_version={version:{val:1713924048510522000, v:0}, total_part_count:3, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=3487720, in_cluster_service=true, cluster_version={val:1713924048598120000, v:0}, min_cluster_version={val:1713924048598120000, v:0}, max_cluster_version={val:1713924048598120000, v:0}, get_cluster_version_err=0, cluster_version_delta=3400122, cluster_service_master="10.0.0.3:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=322532, succ_cluster_heartbeat_count=107, cluster_heartbeat_interval=50000, local_cluster_version={val:1713923825152731000, v:0}, local_cluster_delta=226845511, force_self_check=false, weak_read_refresh_interval=100000) [2024-04-24 10:00:51.998563] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD2A3-0-0] [lt=76] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=26, this=0x14b0c0e2b6a0, extra_info="palf_id:1001") [2024-04-24 10:00:51.998741] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD2A3-0-0] [lt=156] [PALF STAT LOG LIFE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=728360, this=0x14b0c0e2b8d0, extra_info="palf_id:1001") [2024-04-24 10:00:51.998909] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD2A3-0-0] [lt=111] [PALF STAT LOG TASK TIME](palf_id=1001, self="10.0.0.3:2882", total_slide_log_cnt=5, avg_log_gen_to_freeze_time=38149, avg_log_gen_to_submit_time=38334, avg_log_submit_to_flush_time=485395, avg_log_submit_to_slide_time=690025) [2024-04-24 10:00:51.998965] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=164][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:0, v:0}, log_ts:{val:1713924051274688001, v:0}, lsn:{lsn:88128808}, submit_ts:1713924051285620}, this:0x14b055e308a8, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1001}, ctx:0x14b055e2e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():2, range_array_:[{register_no_:1, has_submitted_:true, has_synced_:false, type:3, data_.length():159}, {register_no_:2, has_submitted_:true, has_synced_:false, type:1, data_.length():67}]}, cb_arg_array_:[{log_type:0x4, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=713324, ctx_lock_wait_time=8) [2024-04-24 10:00:51.999007] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD2A3-0-0] [lt=92][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=330737, log_task={header:{begin_lsn:{lsn:88127737}, end_lsn:{lsn:88129312}, log_id:330737, min_scn:{val:1713924051274688000, v:0}, max_scn:{val:1713924051278217001, v:0}, data_len:1519, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:88127187}, data_checksum:470683898, accum_checksum:1744949970, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924051275928, freeze_ts:1713924051443755, submit_ts:1713924051443929, flushed_ts:1713924051980324, gen_to_freeze cost time:167827, gen_to_submit cost time:168001, submit_to_flush cost time:536395}, fs_cb_begin_ts=1713924051998509, log_life_time=722581) [2024-04-24 10:00:51.999007] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2BFC8F2BD-0-0] [lt=97][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051274688000, v:0}, log_ts:{val:1713924051274688000, v:0}, lsn:{lsn:88127793}, submit_ts:1713924051274688}, this:0x14b148703c40, is_inited_:true, trans_id:{txid:327207}, ls_id:{id:1001}, ctx:0x14b148700cd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=724305, ctx_lock_wait_time=8) [2024-04-24 10:00:51.999076] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2C0D8F35E-0-0] [lt=171][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051278217001, v:0}, log_ts:{val:1713924051278217001, v:0}, lsn:{lsn:88129157}, submit_ts:1713924051286811}, this:0x14b055e30af0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1001}, ctx:0x14b055e2e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x40, arg:null}, {log_type:0x80, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=712248, ctx_lock_wait_time=9) [2024-04-24 10:00:51.999141] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2720) [861318][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4002] get invalid Ethernet speed, use default(devname="eth0") [2024-04-24 10:00:51.999249] WDIAG [SERVER] runTimerTask (ob_server.cpp:3239) [861318][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4002] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4002, ret="OB_INVALID_ARGUMENT") [2024-04-24 10:00:51.999274] INFO [MDS.EVENT]try_decline_rec_scn (mds_table_base.cpp:250) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2C0D8F35E-0-0] [lt=118] DECLINE_REC_SCN(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:1152921504606847582}}, event={alloc:null, timestamp:"2024-04-24 10:00:51.999261", event:"DECLINE_REC_SCN", info_str:"{val:4611686018427387903, v:0} -> {val:1713924051274688001, v:0}", unit_id:255, key_str:"", writer_type:0, writer_id:0, seq_no:0, redo_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, trans_version:{val:18446744073709551615, v:3}, node_type:0, state:5}) [2024-04-24 10:00:51.999465] INFO [MDS.EVENT]on_redo_ (mds_node.ipp:150) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2C0D8F35E-0-0] [lt=129] ON_REDO(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:1152921504606847582}}, event={alloc:null, timestamp:"2024-04-24 10:00:51.999430", event:"ON_REDO", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176911, seq_no:0, redo_scn:{val:1713924051274688001, v:0}, end_scn:{val:4611686018427387903, v:0}, trans_version:{val:0, v:0}, node_type:1, state:1}) [2024-04-24 10:00:51.999634] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2C0D8F35E-0-0] [lt=145] notify MDS(ret=0, trans_id_={txid:176911}, ls_id_={id:1001}, notify_type=1, log_ts={val:1713924051274688001, v:0}, notify_array.count()=2, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():159}, {register_no_:2, has_submitted_:true, has_synced_:true, type:1, data_.length():67}], total_time=0) [2024-04-24 10:00:51.999815] INFO [MDS.EVENT]on_prepare_ (mds_node.ipp:202) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2C0D8F35E-0-0] [lt=119] ON_PREPARE(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:1152921504606847582}}, event={alloc:null, timestamp:"2024-04-24 10:00:51.999801", event:"ON_PREPARE", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176911, seq_no:0, redo_scn:{val:1713924051274688001, v:0}, end_scn:{val:4611686018427387903, v:0}, trans_version:{val:0, v:0}, node_type:1, state:2}) [2024-04-24 10:00:51.999986] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2C0D8F35E-0-0] [lt=150] notify MDS(ret=0, trans_id_={txid:176911}, ls_id_={id:1001}, notify_type=3, log_ts={val:1713924051278217001, v:0}, notify_array.count()=2, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():159}, {register_no_:2, has_submitted_:true, has_synced_:true, type:1, data_.length():67}], total_time=0) [2024-04-24 10:00:52.000199] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2C0D8F35E-0-0] [lt=107] rpc statistics(total_trans_msg_count=6, total_batch_msg_count=5) [2024-04-24 10:00:52.001307] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [862447][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=95] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:52.001406] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=81][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051549636000, v:0}, log_ts:{val:1713924051549636000, v:0}, lsn:{lsn:314136577}, submit_ts:1713924051549636}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568577}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=451757, ctx_lock_wait_time=6) [2024-04-24 10:00:52.001602] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2BFC8F2BF-0-0] [lt=103][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:0, v:0}, log_ts:{val:1713924051274688001, v:0}, lsn:{lsn:79627605}, submit_ts:1713924051284482}, this:0x14b055e1c8a8, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():4, range_array_:[{register_no_:1, has_submitted_:true, has_synced_:false, type:1, data_.length():76}, {register_no_:2, has_submitted_:true, has_synced_:false, type:1, data_.length():76}, {register_no_:3, has_submitted_:true, has_synced_:false, type:1, data_.length():67}, {register_no_:4, has_submitted_:true, has_synced_:false, type:6, data_.length():1099}]}, cb_arg_array_:[{log_type:0x4, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=717041, ctx_lock_wait_time=69) [2024-04-24 10:00:52.001733] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=186][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051825760000, v:0}, log_ts:{val:1713924051825760000, v:0}, lsn:{lsn:314137063}, submit_ts:1713924051825760}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:568578}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=175956, ctx_lock_wait_time=10) [2024-04-24 10:00:52.001774] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2BFC8F2BF-0-0] [lt=124] notify MDS(ret=0, trans_id_={txid:176911}, ls_id_={id:1}, notify_type=1, log_ts={val:1713924051274688001, v:0}, notify_array.count()=4, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:true, type:1, data_.length():76}, {register_no_:2, has_submitted_:true, has_synced_:true, type:1, data_.length():76}, {register_no_:3, has_submitted_:true, has_synced_:true, type:1, data_.length():67}, {register_no_:4, has_submitted_:true, has_synced_:true, type:6, data_.length():1099}], total_time=0) [2024-04-24 10:00:52.001969] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:681) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2BFC8F2BF-0-0] [lt=163] update id meta(ret=0, service_type=0, limited_id=1713924068572929000, latest_scn={val:1713924051925376001, v:0}, *this={tenant_id:1004, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895205050766000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924044904071000, v:0}, tablet_change_checkpoint_scn:{val:1713877445609286000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924068572929000, latest_log_ts:{val:1713924051925376001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713919200442801001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}) [2024-04-24 10:00:52.002114] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=192][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051474521000, v:0}, log_ts:{val:1713924051474521000, v:0}, lsn:{lsn:314135992}, submit_ts:1713924051474521}, this:0x14b15a160500, is_inited_:true, trans_id:{txid:391373}, ls_id:{id:1}, ctx:0x14b15a15d590, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=527570, ctx_lock_wait_time=10) [2024-04-24 10:00:52.002124] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:189) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2BFC8F2BF-0-0] [lt=132] handle submit callback(service_type_=0, success=true, used_ts=75318, log_ts={val:1713924051925376001, v:0}, limited_id=1713924068572929000) [2024-04-24 10:00:52.002228] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:561) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2BFC8F2BF-0-0] [lt=93] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2024-04-24 10:00:52.002334] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:599) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2BFC8F2BF-0-0] [lt=92] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2024-04-24 10:00:52.002632] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=94][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051284482000, v:0}, log_ts:{val:1713924051284482000, v:0}, lsn:{lsn:79629066}, submit_ts:1713924051285620}, this:0x14b055e1caf0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x80, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=716999, ctx_lock_wait_time=7) [2024-04-24 10:00:52.002687] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=122] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2024-04-24 10:00:52.002813] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=116] notify MDS(ret=0, trans_id_={txid:176911}, ls_id_={id:1}, notify_type=3, log_ts={val:1713924051284482000, v:0}, notify_array.count()=4, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:true, type:1, data_.length():76}, {register_no_:2, has_submitted_:true, has_synced_:true, type:1, data_.length():76}, {register_no_:3, has_submitted_:true, has_synced_:true, type:1, data_.length():67}, {register_no_:4, has_submitted_:true, has_synced_:true, type:6, data_.length():1099}], total_time=0) [2024-04-24 10:00:52.002971] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2C0D8F35E-0-0] [lt=81][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051755477000, v:0}, log_ts:{val:1713924051755477000, v:0}, lsn:{lsn:79638192}, submit_ts:1713924051755477}, this:0x14b148006200, is_inited_:true, trans_id:{txid:327210}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=247477, ctx_lock_wait_time=10) [2024-04-24 10:00:52.008546] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2C20C019D-0-0] [lt=77] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=13, this=0x14b0c0ef7470, extra_info="") [2024-04-24 10:00:52.008645] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2C20C019D-0-0] [lt=78] [PALF STAT GROUP LOG INFO](palf_id=1002, self="10.0.0.3:2882", role="FOLLOWER", total_group_log_cnt=4, avg_log_batch_cnt=0, total_group_log_size=1530, avg_group_log_size=382) [2024-04-24 10:00:52.008787] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2C20C019D-0-0] [lt=136] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="10.0.0.3:2882", committed size=3173) [2024-04-24 10:00:52.009013] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2C20C019D-0-0] [lt=144] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=38, this=0x14b0c0ef76a0, extra_info="palf_id:1002") [2024-04-24 10:00:52.009090] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2C20C019D-0-0] [lt=57] [PALF STAT LOG LIFE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=353311, this=0x14b0c0ef78d0, extra_info="palf_id:1002") [2024-04-24 10:00:52.009081] INFO [PALF] read (log_cache.cpp:100) [874560][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=110] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=3173, hit_cnt=3, read_cnt=3, hit rate=1.000000000000000000e+00) [2024-04-24 10:00:52.009168] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2C20C019D-0-0] [lt=70] [PALF STAT LOG TASK TIME](palf_id=1002, self="10.0.0.3:2882", total_slide_log_cnt=3, avg_log_gen_to_freeze_time=20, avg_log_gen_to_submit_time=107, avg_log_submit_to_flush_time=287588, avg_log_submit_to_slide_time=353204) [2024-04-24 10:00:52.009251] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2C20C019D-0-0] [lt=71][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=299920, log_task={header:{begin_lsn:{lsn:40361069}, end_lsn:{lsn:40361906}, log_id:299920, min_scn:{val:1713924051138765000, v:0}, max_scn:{val:1713924051170386000, v:0}, data_len:781, proposal_id:5, prev_lsn:{lsn:40360947}, prev_proposal_id:5, committed_end_lsn:{lsn:40361069}, data_checksum:3902936384, accum_checksum:1226935354, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924051263577, freeze_ts:1713924051263586, submit_ts:1713924051263635, flushed_ts:1713924051979151, gen_to_freeze cost time:9, gen_to_submit cost time:58, submit_to_flush cost time:715516}, fs_cb_begin_ts=1713924052008953, log_life_time=745376) [2024-04-24 10:00:52.009697] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [874558][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=84] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924041880302, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924052008787, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:52.009783] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [874557][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=75] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924035421194, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924052008787, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:52.009841] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=169] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=1508) [2024-04-24 10:00:52.009956] INFO [LIB] stat (utility.h:1197) [874558][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=133] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=172, this=0x14b1312048f8, extra_info="") [2024-04-24 10:00:52.010007] INFO [LIB] stat (utility.h:1197) [874557][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=77] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=573, this=0x14b131204b28, extra_info="") [2024-04-24 10:00:52.010007] INFO [LIB] stat (utility.h:1197) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=128] [PALF STAT APPEND COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=0, this=0x14b147074e50, extra_info="id:1") [2024-04-24 10:00:52.010088] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=69] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.010355] INFO [LIB] stat (utility.h:1197) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=100] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=100, this=0x14b1336eb470, extra_info="") [2024-04-24 10:00:52.010498] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=99] [PALF STAT GROUP LOG INFO](palf_id=1001, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=2, avg_log_batch_cnt=1, total_group_log_size=244, avg_group_log_size=122) [2024-04-24 10:00:52.010614] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=87] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, append size=244) [2024-04-24 10:00:52.010771] INFO [LIB] stat (utility.h:1197) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=115] [PALF STAT APPEND COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=0, this=0x14b134390e50, extra_info="id:1001") [2024-04-24 10:00:52.010964] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=160] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.011173] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=196] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.014272] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=87][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:165105970}], scn_array=[{val:1713924051453521000, v:0}], curr_size=1573, accum_size=1573, time_cost=38552) [2024-04-24 10:00:52.016268] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2326786][T1002_L0_G1][T1002][YB420A000003-000616C2B35ECB5A-0-0] [lt=101] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="10.0.0.3:2882", committed size=3390) [2024-04-24 10:00:52.016417] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000003-000616C2B35ECB5A-0-0] [lt=93] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=65, this=0x14b13adc36a0, extra_info="palf_id:1002") [2024-04-24 10:00:52.016553] INFO [PALF] read (log_cache.cpp:100) [862236][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=77] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=3390, hit_cnt=3, read_cnt=3, hit rate=1.000000000000000000e+00) [2024-04-24 10:00:52.016553] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000003-000616C2B35ECB5A-0-0] [lt=115] [PALF STAT LOG LIFE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=419065, this=0x14b13adc38d0, extra_info="palf_id:1002") [2024-04-24 10:00:52.016820] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2326786][T1002_L0_G1][T1002][YB420A000003-000616C2B35ECB5A-0-0] [lt=255] [PALF STAT LOG TASK TIME](palf_id=1002, self="10.0.0.3:2882", total_slide_log_cnt=3, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=176, avg_log_submit_to_flush_time=351768, avg_log_submit_to_slide_time=418889) [2024-04-24 10:00:52.017074] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000003-000616C2B35ECB5A-0-0] [lt=236][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=301040, log_task={header:{begin_lsn:{lsn:165105848}, end_lsn:{lsn:165105970}, log_id:301040, min_scn:{val:1713924050962740001, v:0}, max_scn:{val:1713924050962740001, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:165104275}, prev_proposal_id:5, committed_end_lsn:{lsn:165104275}, data_checksum:1715074492, accum_checksum:358569972, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924051262089, freeze_ts:1713924051262100, submit_ts:1713924051262185, flushed_ts:1713924051975592, gen_to_freeze cost time:11, gen_to_submit cost time:96, submit_to_flush cost time:713407}, fs_cb_begin_ts=1713924052016371, log_life_time=754282) [2024-04-24 10:00:52.017166] INFO [LIB] stat (utility.h:1197) [862286][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=92] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=523, this=0x14b16c6048f8, extra_info="") [2024-04-24 10:00:52.017308] INFO [LIB] stat (utility.h:1197) [862286][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=126] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=389, this=0x14b16c604b28, extra_info="") [2024-04-24 10:00:52.018696] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=89][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:298527450}], scn_array=[{val:1713924051470594000, v:0}], curr_size=1351, accum_size=1351, time_cost=40430) [2024-04-24 10:00:52.019554] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=63][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051748148000, v:0}, log_ts:{val:1713924051748148000, v:0}, lsn:{lsn:298528091}, submit_ts:1713924051748148}, this:0x14b0fbee5680, is_inited_:true, trans_id:{txid:548739}, ls_id:{id:1}, ctx:0x14b0fbee2710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=271387, ctx_lock_wait_time=10) [2024-04-24 10:00:52.020021] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=104] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924040366660, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924052019917, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:52.020249] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=267][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051470594000, v:0}, log_ts:{val:1713924051470594000, v:0}, lsn:{lsn:298527506}, submit_ts:1713924051470594}, this:0x14b0fc0a5ac0, is_inited_:true, trans_id:{txid:338715}, ls_id:{id:1}, ctx:0x14b0fc0a2b50, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=549618, ctx_lock_wait_time=19) [2024-04-24 10:00:52.020577] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=93] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2024-04-24 10:00:52.020595] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=163][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:40361906}], scn_array=[{val:1713924051288012000, v:0}], curr_size=122, accum_size=122, time_cost=41104) [2024-04-24 10:00:52.022515] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:860) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=100] memtable handle rollback to successfuly(from_seq_no=1713924052019918, to_seq_no=1713924052017400, *this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877473} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=1, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}) [2024-04-24 10:00:52.022756] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:670) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=103] rpc statistics(total_trans_msg_count=3, total_batch_msg_count=0) [2024-04-24 10:00:52.022925] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:144) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=97] transaction rpc statistics(total_rt=60000, total_process=214, avg_rt=279) [2024-04-24 10:00:52.025260] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=90][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67160331}], scn_array=[{val:1713924051680715000, v:0}], curr_size=766, accum_size=766, time_cost=43040) [2024-04-24 10:00:52.025325] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [862230][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=176] Slow write(duration=37856, write_len=4096) [2024-04-24 10:00:52.025361] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=173][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179750746}], scn_array=[{val:1713924051459674001, v:0}], curr_size=604, accum_size=604, time_cost=31368) [2024-04-24 10:00:52.025453] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [862230][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=107] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=101082, offset=22880641}, end_cursor:ObLogCursor{file_id=1, log_id=101083, offset=22880882}, is_inited:true, is_local:false, buf_size:8192, buf:0x14b19ef88050, len:3711, log_data_len:241, seq:101082, flush_finish:false, flush_ret:0}) [2024-04-24 10:00:52.025531] INFO [LIB] stat (utility.h:1197) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=116] [PALF STAT LOG IO WORKER QUEUE SIZE](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1, this=0x14b1945fe1e8, extra_info="log_io_worker_num:1, cb_thread_pool_tg_id:297") [2024-04-24 10:00:52.025482] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=138][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:1002197261}], scn_array=[{val:1713924051925376000, v:0}], curr_size=122, accum_size=122, time_cost=14793) [2024-04-24 10:00:52.026805] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=82][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051680715000, v:0}, log_ts:{val:1713924051680715000, v:0}, lsn:{lsn:67160387}, submit_ts:1713924051680715}, this:0x14b168f7bb00, is_inited_:true, trans_id:{txid:869518}, ls_id:{id:1}, ctx:0x14b168f78b90, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=346066, ctx_lock_wait_time=12) [2024-04-24 10:00:52.030745] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=85][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:79638706}], scn_array=[{val:1713924051925376002, v:0}], curr_size=1601, accum_size=12758, time_cost=25078) [2024-04-24 10:00:52.030745] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=65][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:88129312}], scn_array=[{val:1713924051468262000, v:0}], curr_size=602, accum_size=602, time_cost=48876) [2024-04-24 10:00:52.031586] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2BFC8F2BF-0-0] [lt=94][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924051468262000, v:0}, log_ts:{val:1713924051468262000, v:0}, lsn:{lsn:88129368}, submit_ts:1713924051468262}, this:0x14b1481bc200, is_inited_:true, trans_id:{txid:327208}, ls_id:{id:1001}, ctx:0x14b1481b9290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=563284, ctx_lock_wait_time=26) [2024-04-24 10:00:52.032693] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=71] get number(ret=0, service_type_=0, range=1, base_id=1713924052032393000, start_id=1713924052032393000, end_id=1713924052032393001) [2024-04-24 10:00:52.035743] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=80][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.036452] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=93] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.038917] INFO [PALF] check_parent_health (log_config_mgr.cpp:2744) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=160] re_register_parent reason(palf_id=1002, self="10.0.0.3:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1713924048022649, last_first_register_time_us=1713875588469945, register_time_us=1713924048022649, curr_time_us=1713924052038907) [2024-04-24 10:00:52.039168] INFO [PALF] register_parent_ (log_config_mgr.cpp:2601) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=104] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="10.0.0.3:2882", reason=1, leader="10.0.0.2:2882", register_time_us=1713924052039032, last_submit_register_req_time_us=1713924052039032, register_parent_reason=1) [2024-04-24 10:00:52.039313] INFO [PALF] check_parent_health (log_config_mgr.cpp:2753) [862149][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=131] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="10.0.0.3:2882") [2024-04-24 10:00:52.042856] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2C0D8F35E-0-0] [lt=113] notify MDS(ret=0, trans_id_={txid:176911}, ls_id_={id:1}, notify_type=4, log_ts={val:1713924051925376002, v:0}, notify_array.count()=4, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:true, type:1, data_.length():76}, {register_no_:2, has_submitted_:true, has_synced_:true, type:1, data_.length():76}, {register_no_:3, has_submitted_:true, has_synced_:true, type:1, data_.length():67}, {register_no_:4, has_submitted_:true, has_synced_:true, type:6, data_.length():1099}], total_time=0) [2024-04-24 10:00:52.043119] INFO [STORAGE.TABLELOCK] update_lock_status (ob_lock_memtable.cpp:614) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2C0D8F35E-0-0] [lt=146] out_trans update_lock_status(ret=0, op_info={lock_id:{obj_type:7, obj_id:500864}, lock_mode:4, owner_id:{id:176916}, create_trans_id:{txid:176911}, op_type:2, lock_op_status:1, lock_seq_no:1713924051140132, commit_version:{val:18446744073709551615, v:3}, commit_scn:{val:18446744073709551615, v:3}, create_timestamp:1713924051141329, create_schema_version:-1}, commit_scn={val:1713924051925376002, v:0}, status=2, rec_scn_={val:1713923803347774003, v:0}, ls_id_={id:1}) [2024-04-24 10:00:52.043247] INFO [STORAGE.TABLELOCK] update_lock_status (ob_lock_memtable.cpp:614) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2C0D8F35E-0-0] [lt=116] out_trans update_lock_status(ret=0, op_info={lock_id:{obj_type:8, obj_id:200301}, lock_mode:4, owner_id:{id:176916}, create_trans_id:{txid:176911}, op_type:2, lock_op_status:1, lock_seq_no:1713924051150395, commit_version:{val:18446744073709551615, v:3}, commit_scn:{val:18446744073709551615, v:3}, create_timestamp:1713924051151298, create_schema_version:-1}, commit_scn={val:1713924051925376002, v:0}, status=2, rec_scn_={val:1713923803347774003, v:0}, ls_id_={id:1}) [2024-04-24 10:00:52.043372] INFO [STORAGE.TABLELOCK] update_lock_status (ob_lock_memtable.cpp:614) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2C0D8F35E-0-0] [lt=110] out_trans update_lock_status(ret=0, op_info={lock_id:{obj_type:1, obj_id:500864}, lock_mode:8, owner_id:{id:176916}, create_trans_id:{txid:176911}, op_type:2, lock_op_status:1, lock_seq_no:1713924051162101, commit_version:{val:18446744073709551615, v:3}, commit_scn:{val:18446744073709551615, v:3}, create_timestamp:1713924051162675, create_schema_version:0}, commit_scn={val:1713924051925376002, v:0}, status=2, rec_scn_={val:1713923803347774003, v:0}, ls_id_={id:1}) [2024-04-24 10:00:52.043516] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [874589][T1004_L2_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=104] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924040390421, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924052043088, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:52.045213] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=140] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-04-24 10:00:52.050087] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=200][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179751350}], scn_array=[{val:1713924051990986001, v:0}], curr_size=3083, accum_size=3687, time_cost=24202) [2024-04-24 10:00:52.050410] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=125][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:40362028}], scn_array=[{val:1713924052006424000, v:0}], curr_size=449, accum_size=571, time_cost=29363) [2024-04-24 10:00:52.052567] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=75] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.053587] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=110][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67161097}], scn_array=[{val:1713924051925376000, v:0}], curr_size=122, accum_size=888, time_cost=26503) [2024-04-24 10:00:52.054936] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [874560][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=79] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924021299812, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924052053460, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:52.061492] INFO [LIB] stat (utility.h:1197) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B47F14AF-0-0] [lt=99] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=14, this=0x14b13adc3470, extra_info="") [2024-04-24 10:00:52.061637] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=84] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.061624] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B47F14AF-0-0] [lt=86] [PALF STAT GROUP LOG INFO](palf_id=1002, self="10.0.0.3:2882", role="FOLLOWER", total_group_log_cnt=4, avg_log_batch_cnt=0, total_group_log_size=3390, avg_group_log_size=847) [2024-04-24 10:00:52.062320] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [874398][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=138] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}]) [2024-04-24 10:00:52.063026] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=161][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:79640307}], scn_array=[{val:1713924052045408000, v:0}], curr_size=424, accum_size=13182, time_cost=16954) [2024-04-24 10:00:52.063988] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=1633][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:1002197383}], scn_array=[{val:1713924052032393000, v:0}], curr_size=1572, accum_size=1694, time_cost=30696) [2024-04-24 10:00:52.064018] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=93][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:314137549}], scn_array=[{val:1713924052031099000, v:0}], curr_size=641, accum_size=2254, time_cost=31847) [2024-04-24 10:00:52.064079] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=255][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:298529442}], scn_array=[{val:1713924052029945000, v:0}], curr_size=466, accum_size=2458, time_cost=30081) [2024-04-24 10:00:52.064160] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=161][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:88129914}], scn_array=[{val:1713924051992030000, v:0}], curr_size=2550, accum_size=3152, time_cost=32806) [2024-04-24 10:00:52.065493] INFO [MDS] on_commit (buffer_ctx.h:73) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=141] buffer ctx on commit(this=0x14b045be8060, *this={this:0x14b045be8060, ctx:0x14b053e0b110, ctx:{writer:{writer_type:"TRANS", writer_id:176911}, write_list:list_head:0x14b0d6aadb20, list_tail:0x14b0d6aadb20, list:[this:0x14b0d6aadb20, ls_id:{id:1001}, tablet_id:{id:1152921504606847582}, writer:176911, seq_no:0, redo_scn:{val:1713924051274688001, v:0}, end_scn:MAX, trans_version:MIN, status:SET|TRANS|ON_PREPARE|False, ver_next:0x0, linked:0x0, mds_ctx:0x14b053e0b110, user_data:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}]<->NULL, total_size:1, obj_to_string(state_):"ON_PREPARE"}}) [2024-04-24 10:00:52.065705] INFO [MDS] create_tablet_on_commit_ (ob_tablet_create_delete_mds_user_data.cpp:163) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=146] create tablet commit(this={tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1713924051992030002, v:0}, create_commit_version:1713924051284482000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}) [2024-04-24 10:00:52.065963] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [874495][T1004_OB_SLOG][T1004][Y0-0000000000000000-0-0] [lt=92] Slow write(duration=22784, write_len=4096) [2024-04-24 10:00:52.065978] INFO [MDS.EVENT]on_commit_ (mds_node.ipp:239) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=223] ON_COMMIT(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:1152921504606847582}}, event={alloc:null, timestamp:"2024-04-24 10:00:52.65963", event:"ON_COMMIT", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1713924051992030002, v:0}, create_commit_version:1713924051284482000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176911, seq_no:0, redo_scn:{val:1713924051274688001, v:0}, end_scn:{val:1713924051992030002, v:0}, trans_version:{val:1713924051284482000, v:0}, node_type:1, state:3}) [2024-04-24 10:00:52.066090] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1071) [874489][T1004_FreInfoRe][T1004][Y0-0000000000000000-0-0] [lt=195] update reserved snapshot finished(cost_ts=321358, reserved_snapshot=1713922246871241000) [2024-04-24 10:00:52.066105] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [874495][T1004_OB_SLOG][T1004][Y0-0000000000000000-0-0] [lt=127] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=93236, offset=21597152}, end_cursor:ObLogCursor{file_id=1, log_id=93237, offset=21597393}, is_inited:true, is_local:false, buf_size:8192, buf:0x14b19ef7c050, len:1056, log_data_len:241, seq:93236, flush_finish:false, flush_ret:0}) [2024-04-24 10:00:52.066133] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=112] notify MDS(ret=0, trans_id_={txid:176911}, ls_id_={id:1001}, notify_type=4, log_ts={val:1713924051992030002, v:0}, notify_array.count()=2, notify_array=[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():159}, {register_no_:2, has_submitted_:true, has_synced_:true, type:1, data_.length():67}], total_time=0) [2024-04-24 10:00:52.066250] INFO [STORAGE.TABLELOCK] update_lock_status (ob_lock_memtable.cpp:614) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=98] out_trans update_lock_status(ret=0, op_info={lock_id:{obj_type:2, obj_id:200301}, lock_mode:8, owner_id:{id:176916}, create_trans_id:{txid:176911}, op_type:2, lock_op_status:1, lock_seq_no:1713924051182812, commit_version:{val:18446744073709551615, v:3}, commit_scn:{val:18446744073709551615, v:3}, create_timestamp:1713924051183431, create_schema_version:0}, commit_scn={val:1713924051992030002, v:0}, status=2, rec_scn_={val:1713923808203128000, v:0}, ls_id_={id:1001}) [2024-04-24 10:00:52.068103] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=184][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179754433}], scn_array=[{val:1713924052007215000, v:0}], curr_size=1024, accum_size=4711, time_cost=17635) [2024-04-24 10:00:52.069354] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=141][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.072465] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2366) [861401][pnio1][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=66][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:176911}, sender_ls_id={id:1}, receiver_ls_id={id:9223372036854775807}, msg_type=21, status=-4018, receiver_addr="10.0.0.2:2882", request_id=1713924052067795) [2024-04-24 10:00:52.072565] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:285) [861401][pnio1][T0][YB420A000003-000616C2BFC8F2A2-0-0] [lt=89][errcode=0] trx rpc callback(ret=0, status=-4018, dst="10.0.0.2:2882", result={status:-4018, send_timestamp:1713924052067784, private_data:{val:18446744073709551615, v:3}}) [2024-04-24 10:00:52.074353] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:860) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=105] memtable handle rollback to successfuly(from_seq_no=1713924052073495, to_seq_no=1713924052066014, *this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877474} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=1, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}) [2024-04-24 10:00:52.079374] INFO [STORAGE] exec_wait (ob_tmp_file_cache.cpp:1531) [1556243][T1004_ComTimerT][T1004][Y0-0000000000000000-0-0] [lt=72] succeed to do one round of tmp block io(ret=-4023, loop_nums=0, wait_io_cnt=0, washing_count=0, block_cache_num=0, page_cache_num=0) [2024-04-24 10:00:52.080197] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=90] current clock is (clock_=0, max_seq_=0, mem_limit=213070636, cur_mem_hold=109051904, attr_.tenant_id_=1002) [2024-04-24 10:00:52.080225] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:681) [861715][T1_ReplaySrv7][T1][Y0-0000000000000000-0-0] [lt=69] update id meta(ret=0, service_type=0, limited_id=1713924068636985000, latest_scn={val:1713924051990986001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895204295552002, v:0}, clog_base_lsn:{lsn:67104768}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1713875366171467000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924068636985000, latest_log_ts:{val:1713924051990986001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713918035834957001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1713903554859671002, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}) [2024-04-24 10:00:52.080343] INFO [STORAGE.TRANS] handle_replay_result (ob_id_service.cpp:238) [861715][T1_ReplaySrv7][T1][Y0-0000000000000000-0-0] [lt=110] handle reply result(ret=0, service_type_=0, last_id=1713924068636985000, limited_id=1713924068636985000, log_ts={val:1713924051990986001, v:0}) [2024-04-24 10:00:52.080445] INFO [STORAGE.TRANS] replay (ob_id_service.cpp:211) [861715][T1_ReplaySrv7][T1][Y0-0000000000000000-0-0] [lt=80] ObIDService replay(service_type_=0, ret=0, ls_log={last_id_:1713924068636985000, limit_id_:1713924068636985000}, base_header={version:1, log_type:7, flag:0, need_pre_replay_barrier:false, need_post_replay_barrier:false, is_compressed:false, replay_hint:0}) [2024-04-24 10:00:52.082352] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=133] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.082448] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=82] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.082520] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=64] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.084130] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=77][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:314138190}], scn_array=[{val:1713924052035048000, v:0}], curr_size=468, accum_size=2722, time_cost=19413) [2024-04-24 10:00:52.091985] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [862230][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=89] Slow write(duration=26631, write_len=4096) [2024-04-24 10:00:52.092081] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [862230][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=84] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=101083, offset=22880882}, end_cursor:ObLogCursor{file_id=1, log_id=101084, offset=22881123}, is_inited:true, is_local:false, buf_size:8192, buf:0x14b19efa4050, len:3470, log_data_len:241, seq:101083, flush_finish:false, flush_ret:0}) [2024-04-24 10:00:52.092537] INFO eloop_run (eloop.c:143) [861399][pnio1][T0][Y0-0000000000000000-0-0] [lt=73] PNIO [ratelimit] time: 1713924052092526, bytes: 14676909514, bw: 0.283754 MB/s, add_ts: 1003248, add_bytes: 298504 [2024-04-24 10:00:52.093856] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [862269][T1002_ArcSender][T1002][YB420A000003-000616C2AE88F0E7-0-0] [lt=142][errcode=-4018] try free send task failed(ret=-4018) [2024-04-24 10:00:52.094642] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=109][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:88132464}], scn_array=[{val:1713924052008787001, v:0}], curr_size=157, accum_size=3309, time_cost=25000) [2024-04-24 10:00:52.095599] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=146][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:79640731}], scn_array=[{val:1713924052045408001, v:0}], curr_size=157, accum_size=13339, time_cost=27281) [2024-04-24 10:00:52.095737] INFO [STORAGE.TRANS] del_tx_ctx (ob_trans_ctx_mgr_v4.cpp:1314) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2C6-0-0] [lt=80] can not del a retain ctx(ctx={this:0x14b055e2e050, ref:2, trans_id:{txid:176911}, tenant_id:1004, is_exiting:true, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=0], ctx_create_time:1713924051092564}{ls_id:{id:1001}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:50, upstream:{id:1}, participants:[], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:88128808}], redo_log_no:1, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:3, data_.length():159}, {register_no_:2, has_submitted_:true, has_synced_:true, type:1, data_.length():67}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051278217001, v:0}, trans_type:2, next_log_entry_no:4, max_applied_log_ts:{val:1713924052008787001, v:0}, max_applying_log_ts:{val:1713924052008787001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:0, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:88132520}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88129157}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:18}, is_leaf():true, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e30fc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1001}, ctx:0x14b055e2e050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db8b6030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:2, state:"COMMIT", commit_version:{val:1713924051284482000, v:0}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:1713924051992030002, v:0}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:true}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=0 trans_version={val:1713924051278217001, v:0} commit_version={val:1713924051284482000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=-6233 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=1 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=0, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88129157}}, {id_:{id:1}, offset_:{lsn:79629066}}], upstream_state:70, retain_cause:0, 2pc_role:2, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051182811, block_frozen_memtable:null}) [2024-04-24 10:00:52.096397] INFO [STORAGE.TRANS] del_tx_ctx (ob_trans_ctx_mgr_v4.cpp:1314) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=122] can not del a retain ctx(ctx={this:0x14b055e1a050, ref:2, trans_id:{txid:176911}, tenant_id:1004, is_exiting:true, trans_expired_time:1713925048510213, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924051285335], ctx_create_time:1713924048523054}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:50, upstream:{id:1}, participants:[{id:1}, {id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79627605}], redo_log_no:1, multi_data_source:[{register_no_:1, has_submitted_:true, has_synced_:true, type:1, data_.length():76}, {register_no_:2, has_submitted_:true, has_synced_:true, type:1, data_.length():76}, {register_no_:3, has_submitted_:true, has_synced_:true, type:1, data_.length():67}, {register_no_:4, has_submitted_:true, has_synced_:true, type:6, data_.length():1099}], scheduler:"10.0.0.2:2882", prepare_version:{val:1713924051284482000, v:0}, trans_type:2, next_log_entry_no:4, max_applied_log_ts:{val:1713924052045408001, v:0}, max_applying_log_ts:{val:1713924052045408001, v:0}, max_applying_part_log_no:0, max_submitted_seq_no:1713924051282008, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:79640787}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[{id_:{id:1001}, offset_:{lsn:88129157}}, {id_:{id:1}, offset_:{lsn:79629066}}], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:18}, is_leaf():false, is_root():true, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b055e1cfc0, is_inited_:true, trans_id:{txid:176911}, ls_id:{id:1}, ctx:0x14b055e1a050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176911}, ref_cnt:2, state:"COMMIT", commit_version:{val:1713924051284482000, v:0}, start_scn:{val:1713924051274688001, v:0}, end_scn:{val:1713924051925376002, v:0}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:true}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=22003 min_table_version=0 max_table_version=1713877467338312 trans_version={val:1713924051284482000, v:0} commit_version={val:1713924051284482000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=-6233 tx_status=0 is_readonly=false ref=0 trans_id={txid:176911} ls_id=1 callback_alloc_count=31 callback_free_count=31 checksum=2156181382 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=56 redo_sync_succ_count=56 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=56, slave=0, merge=0, tx_end=56, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[{id_:{id:1001}, offset_:{lsn:88129157}}, {id_:{id:1}, offset_:{lsn:79629066}}], upstream_state:70, retain_cause:0, 2pc_role:0, collected:[], rec_log_ts:{val:1713924051274688001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924051285620, block_frozen_memtable:null}) [2024-04-24 10:00:52.096719] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=76][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:1002198955}], scn_array=[{val:1713924052080794000, v:0}], curr_size=1573, accum_size=3267, time_cost=14278) [2024-04-24 10:00:52.096800] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=83] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1003, min_version={val:1713924050970319000, v:0}, base_version={val:1713924050970319000, v:0}, server_count=2, svr_array=[{addr:"10.0.0.2:2882", version:{val:1713924050970319000, v:0}, valid_part_count:0, total_part_count:1, generate_tstamp:1713924052024524, is_skipped:true}, {addr:"10.0.0.3:2882", version:{val:1713924050970319000, v:0}, valid_part_count:0, total_part_count:1, generate_tstamp:1713924052096618, is_skipped:true}], skip_server_count=2, skip_servers=["10.0.0.2:2882", "10.0.0.3:2882"]) [2024-04-24 10:00:52.096994] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=188] no need to update min/max version(new_version={val:1713924050970319000, v:0}, min_version={val:1713924050970319000, v:0}, max_version={val:1713924050970319000, v:0}) [2024-04-24 10:00:52.098625] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [874526][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=125] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1004, min_version={val:1713924048598120000, v:0}, base_version={val:1713924048598120000, v:0}, server_count=2, svr_array=[{addr:"10.0.0.2:2882", version:{val:1713924050897126001, v:0}, valid_part_count:0, total_part_count:3, generate_tstamp:1713924052043853, is_skipped:true}, {addr:"10.0.0.3:2882", version:{val:1713924048510522000, v:0}, valid_part_count:0, total_part_count:3, generate_tstamp:1713924052098494, is_skipped:true}], skip_server_count=2, skip_servers=["10.0.0.2:2882", "10.0.0.3:2882"]) [2024-04-24 10:00:52.098803] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [874526][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=169] no need to update min/max version(new_version={val:1713924048598120000, v:0}, min_version={val:1713924048598120000, v:0}, max_version={val:1713924048598120000, v:0}) [2024-04-24 10:00:52.099345] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [874414][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=102] LogLoopThread round_cost_time(us)(round_cost_time=111) [2024-04-24 10:00:52.103369] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:860) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=105] memtable handle rollback to successfuly(from_seq_no=1713924052102947, to_seq_no=1713924052099243, *this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877475} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=1, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}) [2024-04-24 10:00:52.106006] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=154] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1001, min_version={val:1713924051825760000, v:0}, base_version={val:1713924051049944000, v:0}, server_count=2, svr_array=[{addr:"10.0.0.3:2882", version:{val:1713924051825760000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1713924052105767, is_skipped:false}, {addr:"10.0.0.2:2882", version:{val:1713924051049944000, v:0}, valid_part_count:0, total_part_count:1, generate_tstamp:1713924052023324, is_skipped:true}], skip_server_count=1, skip_servers=["10.0.0.2:2882"]) [2024-04-24 10:00:52.109323] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=139] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=4345) [2024-04-24 10:00:52.109472] INFO [LIB] stat (utility.h:1197) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=117] [PALF STAT APPEND COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x14b14c97ee50, extra_info="id:1") [2024-04-24 10:00:52.110567] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=74] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.110700] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=112] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.110792] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=85] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.116449] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [862226][T1002_ReqMemEvi][T1002][Y0-0000000000000000-0-0] [lt=91] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2024-04-24 10:00:52.118281] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [862226][T1002_ReqMemEvi][T1002][YB420A000003-000616C2B0292E4C-0-0] [lt=129] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924041274670, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924052118103, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:52.118678] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [862226][T1002_ReqMemEvi][T1002][Y0-0000000000000000-0-0] [lt=102] get number(ret=0, service_type_=0, range=1, base_id=1713924052118103000, start_id=1713924052118103000, end_id=1713924052118103001) [2024-04-24 10:00:52.127810] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [862260][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=117] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1002, min_version={val:1713924051680715000, v:0}, base_version={val:1713924048573312000, v:0}, server_count=2, svr_array=[{addr:"10.0.0.3:2882", version:{val:1713924051680715000, v:0}, valid_part_count:1, total_part_count:3, generate_tstamp:1713924052127642, is_skipped:false}, {addr:"10.0.0.2:2882", version:{val:1713924050922254000, v:0}, valid_part_count:0, total_part_count:3, generate_tstamp:1713924052046068, is_skipped:true}], skip_server_count=1, skip_servers=["10.0.0.2:2882"]) [2024-04-24 10:00:52.128167] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=101] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8899076096, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:52.129615] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [862260][T1002_TenantWea][T1001][YB420A000003-000616C2B18CC97D-0-0] [lt=176] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924041631445, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924052128501, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:52.132478] INFO [LIB] stat (utility.h:1197) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AB2FF33B-0-0] [lt=124] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=14, this=0x14b186b47470, extra_info="") [2024-04-24 10:00:52.132612] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AB2FF33B-0-0] [lt=102] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="FOLLOWER", total_group_log_cnt=5, avg_log_batch_cnt=0, total_group_log_size=6333, avg_group_log_size=1266) [2024-04-24 10:00:52.132722] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AB2FF33B-0-0] [lt=97] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=6805) [2024-04-24 10:00:52.135978] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=203][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.136782] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=85] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.142598] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [874742][T1003_LSArchive][T1003][YB420A000003-000616C2B608F0E7-0-0] [lt=72] gc stale ls task succ [2024-04-24 10:00:52.151968] INFO [PALF] check_parent_health (log_config_mgr.cpp:2744) [861573][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=94] re_register_parent reason(palf_id=1, self="10.0.0.3:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1713924048142522, last_first_register_time_us=1713875184097509, register_time_us=1713924048142522, curr_time_us=1713924052151957) [2024-04-24 10:00:52.152202] INFO [PALF] register_parent_ (log_config_mgr.cpp:2601) [861573][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=130] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1, self="10.0.0.3:2882", reason=3, leader="10.0.0.2:2882", register_time_us=1713924052152108, last_submit_register_req_time_us=1713924052152108, register_parent_reason=3) [2024-04-24 10:00:52.152309] INFO [PALF] check_parent_health (log_config_mgr.cpp:2753) [861573][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=99] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="10.0.0.3:2882") [2024-04-24 10:00:52.152555] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [862402][T1001_TenantMet][T1001][Y0-0000000000000000-0-0] [lt=155] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:769, free_obj_hold_cnt:74, allocator used:3351768, allocator total:3466624}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:72, free_obj_hold_cnt:32, allocator used:6821464, allocator total:6821464}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():154752, full allocator used:1600, full allocator total:220288}, tablets_mem=10508376, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:23, free_obj_hold_cnt:140, allocator used:323392, allocator total:392448}, pending_cnt=0, wait_gc_count=0, tablet count=841) [2024-04-24 10:00:52.152860] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=182] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.162132] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=88] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.169553] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=126][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.182540] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=157] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.183008] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=100] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.183098] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=80] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.197236] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [862434][T1001_CKClogDis][T1001][Y0-0000000000000000-0-0] [lt=126] ====== check clog disk timer task ====== [2024-04-24 10:00:52.197337] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [862434][T1001_CKClogDis][T1001][Y0-0000000000000000-0-0] [lt=88] get_disk_usage(ret=0, capacity(MB):=614, used(MB):=363) [2024-04-24 10:00:52.197537] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [862434][T1001_CKClogDis][T1001][Y0-0000000000000000-0-0] [lt=150] cannot_recycle_log_size statistics(cannot_recycle_log_size=179929122, threshold=193273528, need_update_checkpoint_scn=false) [2024-04-24 10:00:52.201223] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [874526][T1004_TenantWea][T1003][YB420A000003-000616C2B8CCACF2-0-0] [lt=120] current clock is (clock_=0, max_seq_=0, mem_limit=264912214, cur_mem_hold=121634816, attr_.tenant_id_=1003) [2024-04-24 10:00:52.210475] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=298] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.210592] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=103] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.210786] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=188] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.214528] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [2320564][T1004_L0_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=77] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924041730701, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924052213383, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:52.221057] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861495][RpcIO][T0][Y0-0000000000000000-0-0] [lt=87] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:52.221137] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861496][RpcIO][T0][Y0-0000000000000000-0-0] [lt=171] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:52.221313] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861498][RpcIO][T0][Y0-0000000000000000-0-0] [lt=78] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:52.221329] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861497][RpcIO][T0][Y0-0000000000000000-0-0] [lt=174] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:52.222256] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [874754][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=140] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:52.236189] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=106][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.237132] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=176] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.245584] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=111] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-04-24 10:00:52.253015] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=151] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.262003] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=148] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.262700] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B47F14B0-0-0] [lt=116][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=301042, log_task={header:{begin_lsn:{lsn:165107543}, end_lsn:{lsn:165107665}, log_id:301042, min_scn:{val:1713924052017398000, v:0}, max_scn:{val:1713924052017398000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:165105970}, prev_proposal_id:5, committed_end_lsn:{lsn:165107543}, data_checksum:1715074492, accum_checksum:3808726171, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924052061431, freeze_ts:1713924052061440, submit_ts:1713924052061590, flushed_ts:1713924052070609, gen_to_freeze cost time:9, gen_to_submit cost time:159, submit_to_flush cost time:9019}, fs_cb_begin_ts=1713924052262633, log_life_time=201202) [2024-04-24 10:00:52.263441] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [874525][T1004_ReqMemEvi][T1004][Y0-0000000000000000-0-0] [lt=132] sql audit evict task end(request_manager_->get_tenant_id()=1004, evict_high_mem_level=278313880, evict_high_size_level=90000, evict_batch_count=0, elapse_time=13, size_used=80721, mem_used=131023872) [2024-04-24 10:00:52.266529] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2325187][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B9-0-0] [lt=166][errcode=-4023] tx is 2pc logging(this={this:0x14b148003290, ref:3, trans_id:{txid:176928}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924082118853, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052131332], ctx_create_time:1713924052128501}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052128504, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052130828000, v:0}, log_ts:{val:1713924052130828000, v:0}, lsn:{lsn:79641368}, submit_ts:1713924052130828}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176928}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176928}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924052130828000, v:0}, start_scn:{val:1713924052130828000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465782000 trans_version={val:1713924052130828000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176928} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3518230785 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052130828000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052130828, block_frozen_memtable:null}) [2024-04-24 10:00:52.266758] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2325187][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B9-0-0] [lt=216][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b148003290, ref:3, trans_id:{txid:176928}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924082118853, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052131332], ctx_create_time:1713924052128501}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052128504, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052130828000, v:0}, log_ts:{val:1713924052130828000, v:0}, lsn:{lsn:79641368}, submit_ts:1713924052130828}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176928}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176928}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924052130828000, v:0}, start_scn:{val:1713924052130828000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465782000 trans_version={val:1713924052130828000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176928} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3518230785 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052130828000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052130828, block_frozen_memtable:null}) [2024-04-24 10:00:52.266948] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2325187][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B9-0-0] [lt=181][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176928}) [2024-04-24 10:00:52.267024] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2325187][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B9-0-0] [lt=68][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176928}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924052261986, cluster_id:1713874979}, expire_ts:1713924082126684, commit_start_scn:{val:1713924052123200000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:52.267144] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2325187][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04B9-0-0] [lt=92][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176928}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924052261986, cluster_id:1713874979}, expire_ts:1713924082126684, commit_start_scn:{val:1713924052123200000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:52.269297] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E9A-0-0] [lt=123][errcode=-4023] tx is 2pc logging(this={this:0x14b168f2d710, ref:3, trans_id:{txid:877475}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924652101605, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052123538], ctx_create_time:1713924052101605}{ls_id:{id:1001}, session_id:3221998686, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052121925, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052123200000, v:0}, log_ts:{val:1713924052123200000, v:0}, lsn:{lsn:1002200584}, submit_ts:1713924052123200}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877475}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877475}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924052123200000, v:0}, start_scn:{val:1713924052123200000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924052123200000, v:0} commit_version={val:1713924052123200000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877475} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=3695499703 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052123200000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052123200, block_frozen_memtable:null}) [2024-04-24 10:00:52.269477] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E9A-0-0] [lt=173][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b168f2d710, ref:3, trans_id:{txid:877475}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924652101605, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052123538], ctx_create_time:1713924052101605}{ls_id:{id:1001}, session_id:3221998686, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052121925, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052123200000, v:0}, log_ts:{val:1713924052123200000, v:0}, lsn:{lsn:1002200584}, submit_ts:1713924052123200}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877475}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877475}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924052123200000, v:0}, start_scn:{val:1713924052123200000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924052123200000, v:0} commit_version={val:1713924052123200000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877475} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=3695499703 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052123200000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052123200, block_frozen_memtable:null}) [2024-04-24 10:00:52.269671] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E9A-0-0] [lt=185][errcode=-4023] commit fail(ret=-4023, coord={id:1001}, tx_id={txid:877475}) [2024-04-24 10:00:52.269790] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E9A-0-0] [lt=109][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877475}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924052264667, cluster_id:1713874979}, expire_ts:1713924618349599, commit_start_scn:{val:1713924052099241000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:52.269790] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [862437][T1001_LSArchive][T1001][YB420A000003-000616C2AEF8F0E7-0-0] [lt=186] gc stale ls task succ [2024-04-24 10:00:52.269961] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E9A-0-0] [lt=148][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877475}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924052264667, cluster_id:1713874979}, expire_ts:1713924618349599, commit_start_scn:{val:1713924052099241000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:52.270031] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=65][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.270718] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1457) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B39C6515-0-0] [lt=83] receive register_req from acceptor, ignore(self="10.0.0.3:2882", palf_id=1, child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924052265953, keepalive_ts:-1}) [2024-04-24 10:00:52.270832] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1476) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2B39C6515-0-0] [lt=105] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="10.0.0.3:2882", child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924052265953, keepalive_ts:-1}, is_to_leader=true) [2024-04-24 10:00:52.280338] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:262) [862473][T1002_ArcSender][T1002][YB420A000003-000616C2B138F0E7-0-0] [lt=135] ObArchiveSender is running(thread_index=1) [2024-04-24 10:00:52.283096] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=90] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.283263] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=119] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.283347] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=76] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.308722] INFO [SERVER] switch_schema (ob_service.cpp:1313) [861898][T1_L0_G0][T1][YB420A000002-000616C2AD7C034E-0-0] [lt=129] start to switch schema(arg=schema_info:{schema_version:1713924050928872, tenant_id:1004, sequence_id:5497558139159}, force_refresh:true, is_async:false) [2024-04-24 10:00:52.308945] INFO [SERVER] async_refresh_schema (ob_server_schema_updater.cpp:484) [861898][T1_L0_G0][T1][YB420A000002-000616C2AD7C034E-0-0] [lt=92] schedule async refresh schema task(ret=0, ret="OB_SUCCESS", tenant_id=1004, schema_version=1713924050928872) [2024-04-24 10:00:52.309079] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2465) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=78] [REFRESH_SCHEMA] start to refresh and add schema(tenant_ids=[1004]) [2024-04-24 10:00:52.309182] INFO [SHARE.SCHEMA] refresh_tenant_schema (ob_multi_version_schema_service.cpp:2795) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=91] [REFRESH_SCHEMA] start to refresh and add schema by tenant(tenant_id=1004) [2024-04-24 10:00:52.310651] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=123] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.310837] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=114] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.310858] INFO [SHARE.SCHEMA] refresh_schema (ob_server_schema_service.cpp:5426) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=66] [REFRESH_SCHEMA] start to refresh increment schema(current schema_version=1713924046573584, schema_status={tenant_id:1004, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-24 10:00:52.310946] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=102] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.316245] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [861452][AutoLSLoc][T0][YB420A000003-000616C2AAD916E8-0-0] [lt=103] fetch root ls success(local_cluster_id=1713874979, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.2:2882", sql_port:2881, role:1, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:5, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713921656566585, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.3:2882", sql_port:2881, role:2, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["10.0.0.2:2882", "10.0.0.3:2882"]) [2024-04-24 10:00:52.316454] INFO [SHARE.SCHEMA] check_core_schema_change_ (ob_server_schema_service.cpp:6566) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=104] core schema is not changed(ret=0, ret="OB_SUCCESS", schema_status={tenant_id:1004, snapshot_timestamp:-1, readable_schema_version:-1}, core_schema_version=1713877451559520, new_core_schema_version=1713877451559520) [2024-04-24 10:00:52.329001] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=135] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8903270400, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:52.329770] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [862245][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=63] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2024-04-24 10:00:52.332618] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0534-0-0] [lt=114][errcode=-4023] tx is 2pc logging(this={this:0x14b14807aed0, ref:3, trans_id:{txid:176929}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925052083398, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052225549], ctx_create_time:1713924052213383}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052221452, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052225180000, v:0}, log_ts:{val:1713924052225180000, v:0}, lsn:{lsn:79641792}, submit_ts:1713924052225180}, this:0x14b14807de40, is_inited_:true, trans_id:{txid:176929}, ls_id:{id:1}, ctx:0x14b14807aed0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176929}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924052225180000, v:0}, start_scn:{val:1713924052225180000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877454130248 max_table_version=1713877454130248 trans_version={val:1713924052225180000, v:0} commit_version={val:1713924052225180000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176929} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3248666907 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052225180000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052225180, block_frozen_memtable:null}) [2024-04-24 10:00:52.332926] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0534-0-0] [lt=300][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b14807aed0, ref:3, trans_id:{txid:176929}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925052083398, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052225549], ctx_create_time:1713924052213383}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052221452, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052225180000, v:0}, log_ts:{val:1713924052225180000, v:0}, lsn:{lsn:79641792}, submit_ts:1713924052225180}, this:0x14b14807de40, is_inited_:true, trans_id:{txid:176929}, ls_id:{id:1}, ctx:0x14b14807aed0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176929}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924052225180000, v:0}, start_scn:{val:1713924052225180000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877454130248 max_table_version=1713877454130248 trans_version={val:1713924052225180000, v:0} commit_version={val:1713924052225180000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176929} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3248666907 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052225180000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052225180, block_frozen_memtable:null}) [2024-04-24 10:00:52.333051] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0534-0-0] [lt=117][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176929}) [2024-04-24 10:00:52.333112] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0534-0-0] [lt=54][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176929}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924052328061, cluster_id:1713874979}, expire_ts:1713925052083381, commit_start_scn:{val:1713924052215814000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:52.333190] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0534-0-0] [lt=62][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176929}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924052328061, cluster_id:1713874979}, expire_ts:1713925052083381, commit_start_scn:{val:1713924052215814000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:52.334420] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=50] ====== checkpoint timer task ====== [2024-04-24 10:00:52.334589] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=112] get_max_applied_scn(scn={val:1713924052029945000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:7, palf_committed_end_lsn_:{lsn:298529908}, last_check_scn_:{val:1713924052029945001, v:0}, max_applied_cb_scn_:{val:1713924052029945000, v:0}}) [2024-04-24 10:00:52.334744] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=132] get_min_unreplayed_log_info(lsn={lsn:283486487}, scn={val:1713900515126283548, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713921656870314, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:283486487}, next_to_submit_scn_:{val:1713900515126283548, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x14b12a08d000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:283486487}, end_lsn:{lsn:283486487}, read_buf:{buf_len_:8192, buf_:0x14b12a08d000}, block_size:67104768, log_storage_:0x14b0f0aa7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-04-24 10:00:52.336157] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=150] succ to get rec scn(*this={this:0x14b0f2804030, ls_id:{id:1}, tenant_id:1003, state:"L_WORKING", total_tx_ctx_count:2, active_tx_count:2, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1713895200598622000, v:0}, last_push_gc_task_ts_:1713895205297191, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:844}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-24 10:00:52.336280] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=114] tx ctx memtable get rec scn(this={ObITable:{this:0x14b0f0adc080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1713895201189153, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x14b0f0adc080, snapshot_version:{val:1713895201189153, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:1713924052199925000, v:0}) [2024-04-24 10:00:52.336355] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=114][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.336376] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=82] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-04-24 10:00:52.336770] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=105] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1713895315527235001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1713895315527235001, v:0}}, }, *this={this:0x14b0f0a70510, is_inited:true, freezing_scn:{val:1713878494144158000, v:0}, ls:{ls_meta:{tenant_id:1003, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895201246660000, v:0}, clog_base_lsn:{lsn:67104768}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1713877340363486001, v:0}, all_id_meta:{id_meta:[{limited_id:1713924059270392000, latest_log_ts:{val:1713924042654251001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713900544098309001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1713877439572285001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:7, palf_env_:0x14b14219e030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:52.336970] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=181] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x14b0f0a7eb90}, min_rec_scn={val:1713895201246660000, v:0}, ls_id_={id:1}) [2024-04-24 10:00:52.337064] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=81] get rec log scn(service_type_=0, rec_log_ts={val:1713924042654251001, v:0}) [2024-04-24 10:00:52.337150] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=79] get rec log scn(service_type_=1, rec_log_ts={val:1713900544098309001, v:0}) [2024-04-24 10:00:52.337249] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=93] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2024-04-24 10:00:52.337264] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=99] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.337346] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=85] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713895201246660000, v:0}, checkpoint_scn_in_ls_meta={val:1713895201246660000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-04-24 10:00:52.337438] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=82][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, curr_base_lsn={lsn:67104768}, new_base_lsn={lsn:67104768}, lsn={lsn:67104768}) [2024-04-24 10:00:52.337498] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [874739][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=117] ====== check clog disk timer task ====== [2024-04-24 10:00:52.337549] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=101] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:67104768}, ls->get_ls_id()={id:1}) [2024-04-24 10:00:52.337581] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [874739][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=73] get_disk_usage(ret=0, capacity(MB):=3686, used(MB):=348) [2024-04-24 10:00:52.337652] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [874737][T1003_TxCkpt][T1003][Y0-0000000000000000-0-0] [lt=97] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-04-24 10:00:52.337743] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [874739][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=87] cannot_recycle_log_size statistics(cannot_recycle_log_size=231425140, threshold=1159641169, need_update_checkpoint_scn=false) [2024-04-24 10:00:52.346196] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [862131][T1002_Occam][T1002][YB420A000003-000616C2AFD988F4-0-0] [lt=79] for each ls(succ_num=3, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:52.350387] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [874735][T1003_ReqMemEvi][T1003][Y0-0000000000000000-0-0] [lt=95] sql audit evict task end(request_manager_->get_tenant_id()=1003, evict_high_mem_level=19327352, evict_high_size_level=90000, evict_batch_count=0, elapse_time=7, size_used=8237, mem_used=14558208) [2024-04-24 10:00:52.353214] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=86] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.357188] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [874535][T1004_ArcSender][T1004][YB420A000003-000616C2B598F0E7-0-0] [lt=74][errcode=-4018] try free send task failed(ret=-4018) [2024-04-24 10:00:52.362217] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=121] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.370237] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=101][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.372644] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [874741][T1003_TabletShe][T1003][Y0-0000000000000000-0-0] [lt=63] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-04-24 10:00:52.372825] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [874741][T1003_TabletShe][T1003][Y0-0000000000000000-0-0] [lt=136] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=9344) [2024-04-24 10:00:52.377928] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:834) [862160][T1002_BalanceEx][T1002][YB420A000003-000616C2B3D9ADA3-0-0] [lt=97] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2024-04-24 10:00:52.378039] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:171) [862160][T1002_BalanceEx][T1002][YB420A000003-000616C2B3D9ADA3-0-0] [lt=92] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2024-04-24 10:00:52.380159] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=78] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: MICRO_BLOCK_CACHE_IO, mode: read, size: 4096.00, iops: 11.92, rt: 1046.00) [2024-04-24 10:00:52.380257] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=74] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:52.380733] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=75] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 129.00) [2024-04-24 10:00:52.380927] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=126] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:52.381791] INFO [COMMON] print_io_status (ob_io_manager.cpp:1393) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=69] [IO STATUS SYS](tenant_id=1002, io_status=sys_group_name: SLOG_IO, mode: write, size: 4096.00, iops: 1.99, rt: 31763.00) [2024-04-24 10:00:52.381945] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=133] [IO STATUS CONFIG](tenant_id=1002, ref_cnt=2, io_config={group_num_:0, memory_limit_:1073741824, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:1}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:52.382794] INFO [COMMON] print_io_status (ob_io_manager.cpp:1393) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=60] [IO STATUS SYS](tenant_id=1004, io_status=sys_group_name: SLOG_IO, mode: write, size: 4096.00, iops: 1.99, rt: 367001.00) [2024-04-24 10:00:52.382960] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=144] [IO STATUS CONFIG](tenant_id=1004, ref_cnt=2, io_config={group_num_:0, memory_limit_:11596411700, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:2}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-04-24 10:00:52.383036] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=64] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-24 10:00:52.383113] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [861365][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=62] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-24 10:00:52.383286] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=73] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.383384] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=78] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.383543] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=151] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.400168] INFO [PALF] check_parent_health (log_config_mgr.cpp:2744) [874414][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=102] re_register_parent reason(palf_id=1002, self="10.0.0.3:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1713924048390587, last_first_register_time_us=1713877519298378, register_time_us=1713924048390587, curr_time_us=1713924052400157) [2024-04-24 10:00:52.400384] INFO [PALF] register_parent_ (log_config_mgr.cpp:2601) [874414][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=100] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="10.0.0.3:2882", reason=1, leader="10.0.0.2:2882", register_time_us=1713924052400278, last_submit_register_req_time_us=1713924052400278, register_parent_reason=1) [2024-04-24 10:00:52.400509] INFO [PALF] check_parent_health (log_config_mgr.cpp:2753) [874414][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=118] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="10.0.0.3:2882") [2024-04-24 10:00:52.410770] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=94] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.410966] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=166] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.411074] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=98] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.416498] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [861593][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=71] ====== tenant freeze timer task ====== [2024-04-24 10:00:52.418329] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [861702][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=67] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:52.436564] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=87][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.437409] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=91] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.445098] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861506][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=69] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:52.445405] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861507][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=64] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:52.445978] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861509][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=121] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:52.445978] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=145] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-04-24 10:00:52.446332] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861508][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=74] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:52.447336] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [861511][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=162] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:52.447964] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861500][BatchIO][T0][Y0-0000000000000000-0-0] [lt=90] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:52.449178] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861501][BatchIO][T0][Y0-0000000000000000-0-0] [lt=83] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:52.449194] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861502][BatchIO][T0][Y0-0000000000000000-0-0] [lt=96] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:52.449342] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [2319281][T1004_L0_G0][T1004][YB420A000003-000616C2BFC8F2D0-0-0] [lt=57] current clock is (clock_=0, max_seq_=0, mem_limit=3181127926, cur_mem_hold=297795584, attr_.tenant_id_=1004) [2024-04-24 10:00:52.449962] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861503][BatchIO][T0][Y0-0000000000000000-0-0] [lt=68] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:52.450526] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [861504][BatchIO][T0][Y0-0000000000000000-0-0] [lt=78] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:52.450955] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861513][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=81] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:52.452936] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [874628][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=67] LogLoopThread round_cost_time(us)(round_cost_time=27) [2024-04-24 10:00:52.453436] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=65] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.460010] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [874528][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=141] ====== traversal_flush timer task ====== [2024-04-24 10:00:52.460563] INFO [MDS] flush (mds_table_mgr.cpp:195) [874528][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=97] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60379}, rec_scn:{val:1713895222371319002, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1713895222371319003, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1713895222371319005, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1713895222371319006, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1713895222873479001, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1713895222873479002, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1713895222873479003, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1713895222873479005, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1713895222873479006, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1713895223156376001, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1713895223257209001, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1713895223257209002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1713895223257209003, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1713895223359838001, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1713895223359838002, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1713895223359838003, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1713895223460804001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1713895223460804002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1713895223562595001, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1713895223562595002, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1713895223562595004, v:0}}, {tablet_id:{id:378}, rec_scn:{val:1713895223562595005, v:0}}, {tablet_id:{id:101036}, rec_scn:{val:1713895223562595006, v:0}}, {tablet_id:{id:60005}, rec_scn:{val:1713895223562595007, v:0}}, {tablet_id:{id:60130}, rec_scn:{val:1713895223562595008, v:0}}, {tablet_id:{id:101064}, rec_scn:{val:1713895223562595010, v:0}}, {tablet_id:{id:101080}, rec_scn:{val:1713895223562595011, v:0}}, {tablet_id:{id:50424}, rec_scn:{val:1713895223562595012, v:0}}, {tablet_id:{id:197}, rec_scn:{val:1713895223562595013, v:0}}, {tablet_id:{id:50381}, rec_scn:{val:1713895223877482001, v:0}}, {tablet_id:{id:103}, rec_scn:{val:1713895223877482003, v:0}}, {tablet_id:{id:50373}, rec_scn:{val:1713895223877482004, v:0}}, }, max_consequent_callbacked_scn={val:1713924052086135000, v:0}, *this={this:0x14b0efaf8510, is_inited:true, freezing_scn:{val:1713879137201785000, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895205050766000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924044904071000, v:0}, tablet_change_checkpoint_scn:{val:1713877445609286000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924068572929000, latest_log_ts:{val:1713924051925376001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713919200442801001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:5, palf_env_:0x14b176ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:52.461062] INFO [MDS] flush (mds_table_mgr.cpp:195) [874528][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=287] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:12, ordered_tablets:{{tablet_id:{id:1152921504606847570}, rec_scn:{val:1713924005270741002, v:0}}, {tablet_id:{id:1152921504606847571}, rec_scn:{val:1713924009981239003, v:0}}, {tablet_id:{id:1152921504606847572}, rec_scn:{val:1713924014707803001, v:0}}, {tablet_id:{id:200315}, rec_scn:{val:1713924014707803001, v:0}}, {tablet_id:{id:1152921504606847573}, rec_scn:{val:1713924016371113002, v:0}}, {tablet_id:{id:1152921504606847574}, rec_scn:{val:1713924018036186001, v:0}}, {tablet_id:{id:1152921504606847577}, rec_scn:{val:1713924032690543001, v:0}}, {tablet_id:{id:1152921504606847578}, rec_scn:{val:1713924034513325001, v:0}}, {tablet_id:{id:1152921504606847580}, rec_scn:{val:1713924036970344001, v:0}}, {tablet_id:{id:1152921504606847581}, rec_scn:{val:1713924048588043001, v:0}}, {tablet_id:{id:200317}, rec_scn:{val:1713924048588043001, v:0}}, {tablet_id:{id:1152921504606847582}, rec_scn:{val:1713924051274688001, v:0}}, }, max_consequent_callbacked_scn={val:1713924052008787001, v:0}, *this={this:0x14b0c0f06510, is_inited:true, freezing_scn:{val:1713879643147421001, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1713895202258988000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924044904071000, v:0}, tablet_change_checkpoint_scn:{val:1713923988969639000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:7, palf_env_:0x14b176ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:2, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:52.461323] INFO [MDS] flush (mds_table_mgr.cpp:195) [874528][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=162] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:8, ordered_tablets:{{tablet_id:{id:1152921504606847566}, rec_scn:{val:1713923993909177001, v:0}}, {tablet_id:{id:1152921504606847567}, rec_scn:{val:1713924003752491002, v:0}}, {tablet_id:{id:1152921504606847568}, rec_scn:{val:1713924004450773001, v:0}}, {tablet_id:{id:1152921504606847569}, rec_scn:{val:1713924004958544001, v:0}}, {tablet_id:{id:1152921504606847575}, rec_scn:{val:1713924027071226001, v:0}}, {tablet_id:{id:1152921504606847576}, rec_scn:{val:1713924029579575001, v:0}}, {tablet_id:{id:200316}, rec_scn:{val:1713924034106200001, v:0}}, {tablet_id:{id:1152921504606847579}, rec_scn:{val:1713924035882919002, v:0}}, }, max_consequent_callbacked_scn={val:1713924052006424000, v:0}, *this={this:0x14b0c0eb6510, is_inited:true, freezing_scn:{val:1713923772239168001, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1002}, ls_create_status:1, clog_checkpoint_scn:{val:1713923772377160001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924046083050001, v:0}, tablet_change_checkpoint_scn:{val:1713923978662610000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:2, proposal_id:0, palf_env_:0x14b176ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:2, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:52.461461] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [874528][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=121] succeed to traversal_flush(ret=0, ls_cnt=3) [2024-04-24 10:00:52.462399] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=81] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.466458] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [874648][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=121] ====== tenant freeze timer task ====== [2024-04-24 10:00:52.467003] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [874588][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04BA-0-0] [lt=99][errcode=-4023] tx is 2pc logging(this={this:0x14b148003290, ref:3, trans_id:{txid:176928}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924082118853, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052131332], ctx_create_time:1713924052128501}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052128504, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052130828000, v:0}, log_ts:{val:1713924052130828000, v:0}, lsn:{lsn:79641368}, submit_ts:1713924052130828}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176928}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176928}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924052130828000, v:0}, start_scn:{val:1713924052130828000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465782000 trans_version={val:1713924052130828000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176928} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3518230785 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052130828000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052130828, block_frozen_memtable:null}) [2024-04-24 10:00:52.467160] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [874588][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04BA-0-0] [lt=144][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b148003290, ref:3, trans_id:{txid:176928}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924082118853, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052131332], ctx_create_time:1713924052128501}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052128504, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052130828000, v:0}, log_ts:{val:1713924052130828000, v:0}, lsn:{lsn:79641368}, submit_ts:1713924052130828}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176928}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176928}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924052130828000, v:0}, start_scn:{val:1713924052130828000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465782000 trans_version={val:1713924052130828000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176928} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=3518230785 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052130828000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052130828, block_frozen_memtable:null}) [2024-04-24 10:00:52.467302] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [874588][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04BA-0-0] [lt=134][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176928}) [2024-04-24 10:00:52.467366] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [874588][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04BA-0-0] [lt=58][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176928}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924052462361, cluster_id:1713874979}, expire_ts:1713924082126684, commit_start_scn:{val:1713924052123200000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:52.467442] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [874588][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04BA-0-0] [lt=55][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176928}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924052462361, cluster_id:1713874979}, expire_ts:1713924082126684, commit_start_scn:{val:1713924052123200000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:52.469484] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E9B-0-0] [lt=96][errcode=-4023] tx is 2pc logging(this={this:0x14b168f2d710, ref:3, trans_id:{txid:877475}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924652101605, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052123538], ctx_create_time:1713924052101605}{ls_id:{id:1001}, session_id:3221998686, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052121925, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052123200000, v:0}, log_ts:{val:1713924052123200000, v:0}, lsn:{lsn:1002200584}, submit_ts:1713924052123200}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877475}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877475}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924052123200000, v:0}, start_scn:{val:1713924052123200000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924052123200000, v:0} commit_version={val:1713924052123200000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877475} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=3695499703 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052123200000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052123200, block_frozen_memtable:null}) [2024-04-24 10:00:52.469743] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E9B-0-0] [lt=252][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b168f2d710, ref:3, trans_id:{txid:877475}, tenant_id:1002, is_exiting:false, trans_expired_time:1713924652101605, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052123538], ctx_create_time:1713924052101605}{ls_id:{id:1001}, session_id:3221998686, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1001}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052121925, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052123200000, v:0}, log_ts:{val:1713924052123200000, v:0}, lsn:{lsn:1002200584}, submit_ts:1713924052123200}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877475}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b13a6b6030, tx_data_guard_:{tx_data:{tx_id:{txid:877475}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924052123200000, v:0}, start_scn:{val:1713924052123200000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:1713924052123200000, v:0} commit_version={val:1713924052123200000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877475} ls_id=1001 callback_alloc_count=2 callback_free_count=0 checksum=3695499703 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=4, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052123200000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052123200, block_frozen_memtable:null}) [2024-04-24 10:00:52.470062] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E9B-0-0] [lt=312][errcode=-4023] commit fail(ret=-4023, coord={id:1001}, tx_id={txid:877475}) [2024-04-24 10:00:52.470192] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E9B-0-0] [lt=124][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877475}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924052465026, cluster_id:1713874979}, expire_ts:1713924618349599, commit_start_scn:{val:1713924052099241000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:52.470289] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2B3CC6E9B-0-0] [lt=84][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1002, tx_id:{txid:877475}, receiver:{id:1001}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924052465026, cluster_id:1713874979}, expire_ts:1713924618349599, commit_start_scn:{val:1713924052099241000, v:0}, parts:[{id:1001}], app_trace_info:""}) [2024-04-24 10:00:52.470434] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=124][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.479981] INFO [SHARE.SCHEMA] fill_schema_operation (ob_schema_retrieve_utils.ipp:2698) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=92] fill schema operation(schema_operation=schema_version:1713924050928856, tenant_id:1004, database_id:500002, user_id:0, database_name:"", tablegroup_id:18446744073709551615, table_id:500928, operation_type:"OB_DDL_CREATE_INDEX", label_se_policy_id:500928, label_se_component_id:500928, label_se_label_id:500928, label_se_user_level_id:500928, outline_id:500928, udf_name:"", sequence_id:500928, keystore_id:500928, outline_id:500928, udf_name:"", sequence_id:500928, tablespace_id:500928, profile_id:500928, audit_id:500928, grantee_id:0, grantor_id:500002, ddl_stmt_str:"", dblink_id:500928, directory_id:500928) [2024-04-24 10:00:52.480128] INFO [SHARE.SCHEMA] fill_schema_operation (ob_schema_retrieve_utils.ipp:2698) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=101] fill schema operation(schema_operation=schema_version:1713924050928864, tenant_id:1004, database_id:500002, user_id:0, database_name:"", tablegroup_id:18446744073709551615, table_id:500864, operation_type:"OB_DDL_MODIFY_TABLE_SCHEMA_VERSION", label_se_policy_id:500864, label_se_component_id:500864, label_se_label_id:500864, label_se_user_level_id:500864, outline_id:500864, udf_name:"", sequence_id:500864, keystore_id:500864, outline_id:500864, udf_name:"", sequence_id:500864, tablespace_id:500864, profile_id:500864, audit_id:500864, grantee_id:0, grantor_id:500002, ddl_stmt_str:"", dblink_id:500864, directory_id:500864) [2024-04-24 10:00:52.480255] INFO [SHARE.SCHEMA] fill_schema_operation (ob_schema_retrieve_utils.ipp:2698) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=100] fill schema operation(schema_operation=schema_version:1713924050928872, tenant_id:1004, database_id:0, user_id:0, database_name:"", tablegroup_id:0, table_id:0, operation_type:"OB_DDL_END_SIGN", label_se_policy_id:0, label_se_component_id:0, label_se_label_id:0, label_se_user_level_id:0, outline_id:0, udf_name:"", sequence_id:0, keystore_id:0, outline_id:0, udf_name:"", sequence_id:0, tablespace_id:0, profile_id:0, audit_id:0, grantee_id:0, grantor_id:0, ddl_stmt_str:"", dblink_id:0, directory_id:0) [2024-04-24 10:00:52.481851] INFO [SHARE.SCHEMA] replay_log (ob_server_schema_service.cpp:4555) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=99] schema operation(schema_operation=schema_version:1713924050928856, tenant_id:1004, database_id:500002, user_id:0, database_name:"", tablegroup_id:18446744073709551615, table_id:500928, operation_type:"OB_DDL_CREATE_INDEX", label_se_policy_id:500928, label_se_component_id:500928, label_se_label_id:500928, label_se_user_level_id:500928, outline_id:500928, udf_name:"", sequence_id:500928, keystore_id:500928, outline_id:500928, udf_name:"", sequence_id:500928, tablespace_id:500928, profile_id:500928, audit_id:500928, grantee_id:0, grantor_id:500002, ddl_stmt_str:"", dblink_id:500928, directory_id:500928) [2024-04-24 10:00:52.481994] INFO [SHARE.SCHEMA] replay_log (ob_server_schema_service.cpp:4555) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=121] schema operation(schema_operation=schema_version:1713924050928864, tenant_id:1004, database_id:500002, user_id:0, database_name:"", tablegroup_id:18446744073709551615, table_id:500864, operation_type:"OB_DDL_MODIFY_TABLE_SCHEMA_VERSION", label_se_policy_id:500864, label_se_component_id:500864, label_se_label_id:500864, label_se_user_level_id:500864, outline_id:500864, udf_name:"", sequence_id:500864, keystore_id:500864, outline_id:500864, udf_name:"", sequence_id:500864, tablespace_id:500864, profile_id:500864, audit_id:500864, grantee_id:0, grantor_id:500002, ddl_stmt_str:"", dblink_id:500864, directory_id:500864) [2024-04-24 10:00:52.482055] INFO [SHARE.SCHEMA] replay_log (ob_server_schema_service.cpp:4555) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=55] schema operation(schema_operation=schema_version:1713924050928872, tenant_id:1004, database_id:0, user_id:0, database_name:"", tablegroup_id:0, table_id:0, operation_type:"OB_DDL_END_SIGN", label_se_policy_id:0, label_se_component_id:0, label_se_label_id:0, label_se_user_level_id:0, outline_id:0, udf_name:"", sequence_id:0, keystore_id:0, outline_id:0, udf_name:"", sequence_id:0, tablespace_id:0, profile_id:0, audit_id:0, grantee_id:0, grantor_id:0, ddl_stmt_str:"", dblink_id:0, directory_id:0) [2024-04-24 10:00:52.482164] INFO [SHARE.SCHEMA] add_tenant (ob_schema_mgr.cpp:1165) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=72] add tenant schema(ret=0, tenant_id=1004, tenant_schema={tenant_id:1004, schema_version:1713924050928872, tenant_name:"fake_tenant", name_case_mode:-1, read_only:false, primary_zone:"", locality:"", previous_locality:"", compatibility_mode:-1, gmt_modified:0, drop_tenant_time:0, status:0, in_recyclebin:false, arbitration_service_status:{status:3}}) [2024-04-24 10:00:52.482237] INFO [SHARE.SCHEMA] get_batch_users (ob_schema_service_sql_impl.cpp:3177) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=63] get batch users(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.482345] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3950) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=100] get batch users success(schema_keys=[]) [2024-04-24 10:00:52.482424] INFO [SHARE.SCHEMA] get_batch_databases (ob_schema_service_sql_impl.cpp:3178) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=72] get batch databases(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.482485] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3951) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=55] get batch databases success(schema_keys=[]) [2024-04-24 10:00:52.482545] INFO [SHARE.SCHEMA] get_batch_tablegroups (ob_schema_service_sql_impl.cpp:3179) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=52] get batch tablegroups(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.482617] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3952) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=66] get batch tablegroups success(schema_keys=[]) [2024-04-24 10:00:52.482746] INFO [SHARE.SCHEMA] get_batch_tables (ob_schema_service_sql_impl.cpp:3124) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=100] get batch tables(schema_version=1713924050928872, schema_keys=[{tenant_id:1004, user_id:18446744073709551615, database_id:18446744073709551615, tablegroup_id:18446744073709551615, table_id:500928, outline_id:500928, routine_name:"", routine_id:500928, database_name:"", table_name:"", schema_version:1713924050928856, synonym_id:500928, package_id:500928, trigger_id:500928, sequence_id:500928, sequence_name:"", udf_name:"", udt_id:500928, keystore_id:500928, keystore_name:"", label_se_policy_id:500928, label_se_component_id:500928, label_se_label_id:500928, label_se_user_level_id:500928, tablespace_id:500928, tablespace_name:"", profile_id:500928, audit_id:500928, grantee_id:18446744073709551615, grantor_id:18446744073709551615, col_id:18446744073709551615, obj_type:18446744073709551615, dblink_id:500928, directory_id:500928, context_id:500928, mock_fk_parent_table_id:500928, rls_policy_id:500928, rls_group_id:500928, rls_context_id:500928, routine_type:500928}, {tenant_id:1004, user_id:18446744073709551615, database_id:18446744073709551615, tablegroup_id:18446744073709551615, table_id:500864, outline_id:500864, routine_name:"", routine_id:500864, database_name:"", table_name:"", schema_version:1713924050928864, synonym_id:500864, package_id:500864, trigger_id:500864, sequence_id:500864, sequence_name:"", udf_name:"", udt_id:500864, keystore_id:500864, keystore_name:"", label_se_policy_id:500864, label_se_component_id:500864, label_se_label_id:500864, label_se_user_level_id:500864, tablespace_id:500864, tablespace_name:"", profile_id:500864, audit_id:500864, grantee_id:18446744073709551615, grantor_id:18446744073709551615, col_id:18446744073709551615, obj_type:18446744073709551615, dblink_id:500864, directory_id:500864, context_id:500864, mock_fk_parent_table_id:500864, rls_policy_id:500864, rls_group_id:500864, rls_context_id:500864, routine_type:500864}]) [2024-04-24 10:00:52.483426] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=70] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.483527] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=87] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.483607] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=73] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.484159] INFO [SHARE.SCHEMA] retrieve_table_schema (ob_schema_retrieve_utils.ipp:107) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=88] retrieve table schema(ret=0, ret="OB_SUCCESS", table_schema={tenant_id:1004, database_id:500002, tablegroup_id:18446744073709551615, table_id:500928, association_table_id:18446744073709551615, in_offline_ddl_white_list:false, table_name:"__idx_500864_idx_topUserId", session_id:0, index_type:1, table_type:5, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, tablespace_id:18446744073709551615, data_table_id:500864, name_casemode:-1, schema_version:1713924050928856, part_level:0, part_option:{part_func_type:0, part_func_expr:"", part_num:1, auto_part:false, auto_part_size:0}, sub_part_option:{part_func_type:0, part_func_expr:"", part_num:0, auto_part:false, auto_part_size:0}, partition_num:0, def_subpartition_num:0, partition_array:null, def_subpartition_array:null, hidden_partition_array:null, index_status:1, duplicate_scope:0, encryption:"", encrypt_key:"", master_key_id:18446744073709551615, sub_part_template_flags:0, get_tablet_id():{id:1152921504606847582}, max_dependency_version:-1, object_status:1, is_force_view:false, truncate_version:-1}, is_deleted=false) [2024-04-24 10:00:52.484347] INFO [SHARE.SCHEMA] retrieve_table_schema (ob_schema_retrieve_utils.ipp:107) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=103] retrieve table schema(ret=0, ret="OB_SUCCESS", table_schema={tenant_id:1004, database_id:500002, tablegroup_id:18446744073709551615, table_id:500864, association_table_id:18446744073709551615, in_offline_ddl_white_list:false, table_name:"gs_agent_user_map", session_id:0, index_type:0, table_type:3, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, tablespace_id:18446744073709551615, data_table_id:0, name_casemode:-1, schema_version:1713924050928864, part_level:0, part_option:{part_func_type:0, part_func_expr:"", part_num:1, auto_part:false, auto_part_size:0}, sub_part_option:{part_func_type:0, part_func_expr:"", part_num:0, auto_part:false, auto_part_size:0}, partition_num:0, def_subpartition_num:0, partition_array:null, def_subpartition_array:null, hidden_partition_array:null, index_status:1, duplicate_scope:0, encryption:"", encrypt_key:"", master_key_id:18446744073709551615, sub_part_template_flags:0, get_tablet_id():{id:200301}, max_dependency_version:-1, object_status:1, is_force_view:false, truncate_version:-1}, is_deleted=false) [2024-04-24 10:00:52.484804] INFO [SHARE.SCHEMA] retrieve_table_schema (ob_schema_retrieve_utils.ipp:120) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=123] retrieve table schema array(tenant_id=1004) [2024-04-24 10:00:52.486217] INFO [SHARE.SCHEMA] retrieve_simple_foreign_key_info (ob_schema_retrieve_utils.ipp:4794) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=132] retrieve simple foreign key info(tenant_id=1004) [2024-04-24 10:00:52.487699] INFO [SHARE.SCHEMA] retrieve_simple_constraint_info (ob_schema_retrieve_utils.ipp:4925) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=88] retrieve simple constraint info(tenant_id=1004) [2024-04-24 10:00:52.487916] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3953) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=81] get batch tables success(schema_keys=[{tenant_id:1004, user_id:18446744073709551615, database_id:18446744073709551615, tablegroup_id:18446744073709551615, table_id:500928, outline_id:500928, routine_name:"", routine_id:500928, database_name:"", table_name:"", schema_version:1713924050928856, synonym_id:500928, package_id:500928, trigger_id:500928, sequence_id:500928, sequence_name:"", udf_name:"", udt_id:500928, keystore_id:500928, keystore_name:"", label_se_policy_id:500928, label_se_component_id:500928, label_se_label_id:500928, label_se_user_level_id:500928, tablespace_id:500928, tablespace_name:"", profile_id:500928, audit_id:500928, grantee_id:18446744073709551615, grantor_id:18446744073709551615, col_id:18446744073709551615, obj_type:18446744073709551615, dblink_id:500928, directory_id:500928, context_id:500928, mock_fk_parent_table_id:500928, rls_policy_id:500928, rls_group_id:500928, rls_context_id:500928, routine_type:500928}, {tenant_id:1004, user_id:18446744073709551615, database_id:18446744073709551615, tablegroup_id:18446744073709551615, table_id:500864, outline_id:500864, routine_name:"", routine_id:500864, database_name:"", table_name:"", schema_version:1713924050928864, synonym_id:500864, package_id:500864, trigger_id:500864, sequence_id:500864, sequence_name:"", udf_name:"", udt_id:500864, keystore_id:500864, keystore_name:"", label_se_policy_id:500864, label_se_component_id:500864, label_se_label_id:500864, label_se_user_level_id:500864, tablespace_id:500864, tablespace_name:"", profile_id:500864, audit_id:500864, grantee_id:18446744073709551615, grantor_id:18446744073709551615, col_id:18446744073709551615, obj_type:18446744073709551615, dblink_id:500864, directory_id:500864, context_id:500864, mock_fk_parent_table_id:500864, rls_policy_id:500864, rls_group_id:500864, rls_context_id:500864, routine_type:500864}]) [2024-04-24 10:00:52.488014] INFO [SHARE.SCHEMA] get_batch_outlines (ob_schema_service_sql_impl.cpp:3183) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=86] get batch outlines(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.488112] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3954) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=76] get batch outlines success(schema_keys=[]) [2024-04-24 10:00:52.488189] INFO [SHARE.SCHEMA] get_batch_routines (ob_schema_service_sql_impl.cpp:3185) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=71] get batch routines(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.488269] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3955) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=73] get batch routines success(schema_keys=[]) [2024-04-24 10:00:52.488372] INFO [SHARE.SCHEMA] get_batch_packages (ob_schema_service_sql_impl.cpp:3186) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=90] get batch packages(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.488492] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3956) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=105] get batch packages success(schema_keys=[]) [2024-04-24 10:00:52.488599] INFO [SHARE.SCHEMA] get_batch_triggers (ob_schema_service_sql_impl.cpp:3187) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=98] get batch triggers(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.488717] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3957) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=110] get batch triggers success(schema_keys=[]) [2024-04-24 10:00:52.488803] INFO [SHARE.SCHEMA] get_batch_db_privs (ob_schema_service_sql_impl.cpp:3180) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=74] get batch db_privs(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.488923] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3958) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=107] get batch db_privs success(schema_keys=[]) [2024-04-24 10:00:52.488986] INFO [SHARE.SCHEMA] get_batch_table_privs (ob_schema_service_sql_impl.cpp:3181) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=57] get batch table_privs(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.489053] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3959) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=60] get batch table_privs success(schema_keys=[]) [2024-04-24 10:00:52.489127] INFO [SHARE.SCHEMA] get_batch_routine_privs (ob_schema_service_sql_impl.cpp:3182) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=60] get batch routine_privs(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.489233] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3960) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=94] get batch routine_privs success(schema_keys=[]) [2024-04-24 10:00:52.489306] INFO [SHARE.SCHEMA] get_batch_synonyms (ob_schema_service_sql_impl.cpp:3184) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=60] get batch synonyms(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.489408] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3961) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=89] get batch synonyms success(schema_keys=[]) [2024-04-24 10:00:52.489475] INFO [SHARE.SCHEMA] get_batch_udfs (ob_schema_service_sql_impl.cpp:3188) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=61] get batch udfs(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.489593] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3962) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=108] get batch udfs success(schema_keys=[]) [2024-04-24 10:00:52.489706] INFO [SHARE.SCHEMA] get_batch_udts (ob_schema_service_sql_impl.cpp:3189) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=100] get batch udts(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.489807] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3963) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=94] get batch udts success(schema_keys=[]) [2024-04-24 10:00:52.489923] INFO [SHARE.SCHEMA] get_batch_sequences (ob_schema_service_sql_impl.cpp:3190) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=107] get batch sequences(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.490019] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3964) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=77] get batch sequences success(schema_keys=[]) [2024-04-24 10:00:52.490096] INFO [SHARE.SCHEMA] get_batch_keystores (ob_schema_service_sql_impl.cpp:3191) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=71] get batch keystores(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.490165] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3965) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=60] get batch keystores success(schema_keys=[]) [2024-04-24 10:00:52.490238] INFO [SHARE.SCHEMA] get_batch_label_se_policys (ob_schema_service_sql_impl.cpp:3192) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=64] get batch label_se_policys(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.490300] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3966) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=56] get batch label_se_policys success(schema_keys=[]) [2024-04-24 10:00:52.490377] INFO [SHARE.SCHEMA] get_batch_label_se_components (ob_schema_service_sql_impl.cpp:3193) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=69] get batch label_se_components(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.490453] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3967) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=69] get batch label_se_components success(schema_keys=[]) [2024-04-24 10:00:52.490516] INFO [SHARE.SCHEMA] get_batch_label_se_labels (ob_schema_service_sql_impl.cpp:3194) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=52] get batch label_se_labels(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.490622] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3968) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=93] get batch label_se_labels success(schema_keys=[]) [2024-04-24 10:00:52.490694] INFO [SHARE.SCHEMA] get_batch_label_se_user_levels (ob_schema_service_sql_impl.cpp:3195) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=66] get batch label_se_user_levels(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.490814] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3969) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=113] get batch label_se_user_levels success(schema_keys=[]) [2024-04-24 10:00:52.490939] INFO [SHARE.SCHEMA] get_batch_tablespaces (ob_schema_service_sql_impl.cpp:3196) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=116] get batch tablespaces(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.491008] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3970) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=57] get batch tablespaces success(schema_keys=[]) [2024-04-24 10:00:52.491091] INFO [SHARE.SCHEMA] get_batch_profiles (ob_schema_service_sql_impl.cpp:3197) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=76] get batch profiles(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.491184] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3971) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=85] get batch profiles success(schema_keys=[]) [2024-04-24 10:00:52.491249] INFO [SHARE.SCHEMA] get_batch_audits (ob_schema_service_sql_impl.cpp:3198) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=56] get batch audits(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.491313] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3972) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=56] get batch audits success(schema_keys=[]) [2024-04-24 10:00:52.491390] INFO [SHARE.SCHEMA] get_batch_sys_privs (ob_schema_service_sql_impl.cpp:3199) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=68] get batch sys_privs(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.491462] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3973) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=65] get batch sys_privs success(schema_keys=[]) [2024-04-24 10:00:52.491522] INFO [SHARE.SCHEMA] get_batch_obj_privs (ob_schema_service_sql_impl.cpp:3200) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=54] get batch obj_privs(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.491605] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3974) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=74] get batch obj_privs success(schema_keys=[]) [2024-04-24 10:00:52.491687] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3981) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=73] get batch sys_variables success(schema_keys=[]) [2024-04-24 10:00:52.491762] INFO [SHARE.SCHEMA] get_batch_dblinks (ob_schema_service_sql_impl.cpp:3201) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=66] get batch dblinks(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.491852] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3982) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=81] get batch dblinks success(schema_keys=[]) [2024-04-24 10:00:52.492011] INFO [SHARE.SCHEMA] get_batch_directorys (ob_schema_service_sql_impl.cpp:3202) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=151] get batch directorys(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.492099] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3983) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=75] get batch directorys success(schema_keys=[]) [2024-04-24 10:00:52.492204] INFO [SHARE.SCHEMA] get_batch_contexts (ob_schema_service_sql_impl.cpp:3203) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=94] get batch contexts(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.492294] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3984) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=81] get batch contexts success(schema_keys=[]) [2024-04-24 10:00:52.492370] INFO [SHARE.SCHEMA] get_batch_mock_fk_parent_tables (ob_schema_service_sql_impl.cpp:3204) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=68] get batch mock_fk_parent_tables(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.492436] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3985) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=58] get batch mock_fk_parent_tables success(schema_keys=[]) [2024-04-24 10:00:52.492527] INFO [SHARE.SCHEMA] get_batch_rls_policys (ob_schema_service_sql_impl.cpp:3205) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=73] get batch rls_policys(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.492624] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3986) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=69] get batch rls_policys success(schema_keys=[]) [2024-04-24 10:00:52.492711] INFO [SHARE.SCHEMA] get_batch_rls_groups (ob_schema_service_sql_impl.cpp:3206) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=80] get batch rls_groups(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.492784] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3987) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=62] get batch rls_groups success(schema_keys=[]) [2024-04-24 10:00:52.492862] INFO [SHARE.SCHEMA] get_batch_rls_contexts (ob_schema_service_sql_impl.cpp:3207) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=71] get batch rls_contexts(schema_version=1713924050928872, schema_keys=[]) [2024-04-24 10:00:52.492995] INFO [SHARE.SCHEMA] fetch_increment_schemas (ob_server_schema_service.cpp:3988) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=124] get batch rls_contexts success(schema_keys=[]) [2024-04-24 10:00:52.493065] INFO [SHARE.SCHEMA] get_batch_table_schema (ob_schema_service_sql_impl.cpp:419) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=62] get batch table schema finish(schema_version=1713924050928872, ret=0) [2024-04-24 10:00:52.493134] INFO [SHARE.SCHEMA] apply_tenant_schema_to_cache (ob_server_schema_service.cpp:4164) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=63] del tenants finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.493211] INFO [SHARE.SCHEMA] apply_tenant_schema_to_cache (ob_server_schema_service.cpp:4207) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=68] add tenants finish(schemas=[], ret=0) [2024-04-24 10:00:52.493284] INFO [SHARE.SCHEMA] apply_tenant_schema_to_cache (ob_server_schema_service.cpp:4215) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=65] add tenants finish(schemas=[], ret=0) [2024-04-24 10:00:52.493351] INFO [SHARE.SCHEMA] apply_sys_variable_schema_to_cache (ob_server_schema_service.cpp:4260) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=58] del sys_variables finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.493414] INFO [SHARE.SCHEMA] apply_sys_variable_schema_to_cache (ob_server_schema_service.cpp:4260) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=54] add sys_variables finish(schemas=[], ret=0) [2024-04-24 10:00:52.493514] INFO [SHARE.SCHEMA] apply_user_schema_to_cache (ob_server_schema_service.cpp:4261) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=90] del users finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.493609] INFO [SHARE.SCHEMA] apply_user_schema_to_cache (ob_server_schema_service.cpp:4261) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=87] add users finish(schemas=[], ret=0) [2024-04-24 10:00:52.493751] INFO [SHARE.SCHEMA] apply_database_schema_to_cache (ob_server_schema_service.cpp:4262) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=132] del databases finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.493815] INFO [SHARE.SCHEMA] apply_database_schema_to_cache (ob_server_schema_service.cpp:4262) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=55] add databases finish(schemas=[], ret=0) [2024-04-24 10:00:52.493949] INFO [SHARE.SCHEMA] apply_tablegroup_schema_to_cache (ob_server_schema_service.cpp:4263) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=126] del tablegroups finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.494021] INFO [SHARE.SCHEMA] apply_tablegroup_schema_to_cache (ob_server_schema_service.cpp:4263) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=63] add tablegroups finish(schemas=[], ret=0) [2024-04-24 10:00:52.494100] INFO [SHARE.SCHEMA] apply_table_schema_to_cache (ob_server_schema_service.cpp:4264) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=71] del tables finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.494377] INFO [SHARE.SCHEMA] add_tables (ob_schema_mgr.cpp:2380) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=79] add tables(ret=0, ret="OB_SUCCESS", stage_cost=[15, 37, 13, 20, 18], cost=182) [2024-04-24 10:00:52.494449] INFO [SHARE.SCHEMA] apply_table_schema_to_cache (ob_server_schema_service.cpp:4264) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=64] add tables finish(schemas=[{tenant_id:1004, database_id:500002, tablegroup_id:18446744073709551615, table_id:500928, association_table_id:18446744073709551615, in_offline_ddl_white_list:false, table_name:"__idx_500864_idx_topUserId", session_id:0, index_type:1, table_type:5, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, tablespace_id:18446744073709551615, data_table_id:500864, name_casemode:-1, schema_version:1713924050928856, part_level:0, part_option:{part_func_type:0, part_func_expr:"", part_num:1, auto_part:false, auto_part_size:0}, sub_part_option:{part_func_type:0, part_func_expr:"", part_num:0, auto_part:false, auto_part_size:0}, partition_num:0, def_subpartition_num:0, partition_array:null, def_subpartition_array:null, hidden_partition_array:null, index_status:1, duplicate_scope:0, encryption:"", encrypt_key:"", master_key_id:18446744073709551615, sub_part_template_flags:0, get_tablet_id():{id:1152921504606847582}, max_dependency_version:-1, object_status:1, is_force_view:false, truncate_version:-1}, {tenant_id:1004, database_id:500002, tablegroup_id:18446744073709551615, table_id:500864, association_table_id:18446744073709551615, in_offline_ddl_white_list:false, table_name:"gs_agent_user_map", session_id:0, index_type:0, table_type:3, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, tablespace_id:18446744073709551615, data_table_id:0, name_casemode:-1, schema_version:1713924050928864, part_level:0, part_option:{part_func_type:0, part_func_expr:"", part_num:1, auto_part:false, auto_part_size:0}, sub_part_option:{part_func_type:0, part_func_expr:"", part_num:0, auto_part:false, auto_part_size:0}, partition_num:0, def_subpartition_num:0, partition_array:null, def_subpartition_array:null, hidden_partition_array:null, index_status:1, duplicate_scope:0, encryption:"", encrypt_key:"", master_key_id:18446744073709551615, sub_part_template_flags:0, get_tablet_id():{id:200301}, max_dependency_version:-1, object_status:1, is_force_view:false, truncate_version:-1}], ret=0) [2024-04-24 10:00:52.494567] INFO [SHARE.SCHEMA] apply_outline_schema_to_cache (ob_server_schema_service.cpp:4265) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=111] del outlines finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.494650] INFO [SHARE.SCHEMA] apply_outline_schema_to_cache (ob_server_schema_service.cpp:4265) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=66] add outlines finish(schemas=[], ret=0) [2024-04-24 10:00:52.494753] INFO [SHARE.SCHEMA] apply_routine_schema_to_cache (ob_server_schema_service.cpp:4266) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=71] del routines finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.494923] INFO [SHARE.SCHEMA] apply_routine_schema_to_cache (ob_server_schema_service.cpp:4266) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=103] add routines finish(schemas=[], ret=0) [2024-04-24 10:00:52.495013] INFO [SHARE.SCHEMA] apply_package_schema_to_cache (ob_server_schema_service.cpp:4267) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=81] del packages finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.495075] INFO [SHARE.SCHEMA] apply_package_schema_to_cache (ob_server_schema_service.cpp:4267) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=53] add packages finish(schemas=[], ret=0) [2024-04-24 10:00:52.495149] INFO [SHARE.SCHEMA] apply_trigger_schema_to_cache (ob_server_schema_service.cpp:4268) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=66] del triggers finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.495224] INFO [SHARE.SCHEMA] apply_trigger_schema_to_cache (ob_server_schema_service.cpp:4268) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=67] add triggers finish(schemas=[], ret=0) [2024-04-24 10:00:52.495330] INFO [SHARE.SCHEMA] apply_db_priv_schema_to_cache (ob_server_schema_service.cpp:4269) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=97] del db_privs finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.495407] INFO [SHARE.SCHEMA] apply_db_priv_schema_to_cache (ob_server_schema_service.cpp:4269) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=69] add db_privs finish(schemas=[], ret=0) [2024-04-24 10:00:52.495470] INFO [SHARE.SCHEMA] apply_table_priv_schema_to_cache (ob_server_schema_service.cpp:4270) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=54] del table_privs finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.495534] INFO [SHARE.SCHEMA] apply_table_priv_schema_to_cache (ob_server_schema_service.cpp:4270) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=56] add table_privs finish(schemas=[], ret=0) [2024-04-24 10:00:52.495603] INFO [SHARE.SCHEMA] apply_routine_priv_schema_to_cache (ob_server_schema_service.cpp:4271) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=63] del routine_privs finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.495684] INFO [SHARE.SCHEMA] apply_routine_priv_schema_to_cache (ob_server_schema_service.cpp:4271) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=69] add routine_privs finish(schemas=[], ret=0) [2024-04-24 10:00:52.495754] INFO [SHARE.SCHEMA] apply_synonym_schema_to_cache (ob_server_schema_service.cpp:4272) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=60] del synonyms finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.495848] INFO [SHARE.SCHEMA] apply_synonym_schema_to_cache (ob_server_schema_service.cpp:4272) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=85] add synonyms finish(schemas=[], ret=0) [2024-04-24 10:00:52.495974] INFO [SHARE.SCHEMA] apply_udf_schema_to_cache (ob_server_schema_service.cpp:4273) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=118] del udfs finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.496050] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [874410][T1004_PalfGC][T1004][Y0-0000000000000000-0-0] [lt=89] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):33177, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):33177, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):390, sequence:1}) [2024-04-24 10:00:52.496088] INFO [SHARE.SCHEMA] apply_udf_schema_to_cache (ob_server_schema_service.cpp:4273) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=102] add udfs finish(schemas=[], ret=0) [2024-04-24 10:00:52.496157] INFO [SHARE.SCHEMA] apply_udt_schema_to_cache (ob_server_schema_service.cpp:4274) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=63] del udts finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.496241] INFO [SHARE.SCHEMA] apply_udt_schema_to_cache (ob_server_schema_service.cpp:4274) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=71] add udts finish(schemas=[], ret=0) [2024-04-24 10:00:52.496305] INFO [SHARE.SCHEMA] apply_sequence_schema_to_cache (ob_server_schema_service.cpp:4275) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=55] del sequences finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.496375] INFO [SHARE.SCHEMA] apply_sequence_schema_to_cache (ob_server_schema_service.cpp:4275) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=58] add sequences finish(schemas=[], ret=0) [2024-04-24 10:00:52.496469] INFO [SHARE.SCHEMA] apply_keystore_schema_to_cache (ob_server_schema_service.cpp:4276) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=84] del keystores finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.496532] INFO [SHARE.SCHEMA] apply_keystore_schema_to_cache (ob_server_schema_service.cpp:4276) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=56] add keystores finish(schemas=[], ret=0) [2024-04-24 10:00:52.496591] INFO [SHARE.SCHEMA] apply_label_se_policy_schema_to_cache (ob_server_schema_service.cpp:4277) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=52] del label_se_policys finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.496683] INFO [SHARE.SCHEMA] apply_label_se_policy_schema_to_cache (ob_server_schema_service.cpp:4277) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=62] add label_se_policys finish(schemas=[], ret=0) [2024-04-24 10:00:52.496754] INFO [SHARE.SCHEMA] apply_label_se_component_schema_to_cache (ob_server_schema_service.cpp:4278) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=63] del label_se_components finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.496896] INFO [SHARE.SCHEMA] apply_label_se_component_schema_to_cache (ob_server_schema_service.cpp:4278) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=108] add label_se_components finish(schemas=[], ret=0) [2024-04-24 10:00:52.496967] INFO [SHARE.SCHEMA] apply_label_se_label_schema_to_cache (ob_server_schema_service.cpp:4279) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=65] del label_se_labels finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.497035] INFO [SHARE.SCHEMA] apply_label_se_label_schema_to_cache (ob_server_schema_service.cpp:4279) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=61] add label_se_labels finish(schemas=[], ret=0) [2024-04-24 10:00:52.497100] INFO [SHARE.SCHEMA] apply_label_se_user_level_schema_to_cache (ob_server_schema_service.cpp:4280) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=59] del label_se_user_levels finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.497173] INFO [SHARE.SCHEMA] apply_label_se_user_level_schema_to_cache (ob_server_schema_service.cpp:4280) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=63] add label_se_user_levels finish(schemas=[], ret=0) [2024-04-24 10:00:52.497238] INFO [SHARE.SCHEMA] apply_tablespace_schema_to_cache (ob_server_schema_service.cpp:4281) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=59] del tablespaces finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.497302] INFO [SHARE.SCHEMA] apply_tablespace_schema_to_cache (ob_server_schema_service.cpp:4281) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=58] add tablespaces finish(schemas=[], ret=0) [2024-04-24 10:00:52.497361] INFO [SHARE.SCHEMA] apply_profile_schema_to_cache (ob_server_schema_service.cpp:4282) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=52] del profiles finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.497435] INFO [SHARE.SCHEMA] apply_profile_schema_to_cache (ob_server_schema_service.cpp:4282) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=67] add profiles finish(schemas=[], ret=0) [2024-04-24 10:00:52.497510] INFO [SHARE.SCHEMA] apply_audit_schema_to_cache (ob_server_schema_service.cpp:4283) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=67] del audits finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.497586] INFO [SHARE.SCHEMA] apply_audit_schema_to_cache (ob_server_schema_service.cpp:4283) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=67] add audits finish(schemas=[], ret=0) [2024-04-24 10:00:52.497682] INFO [SHARE.SCHEMA] apply_sys_priv_schema_to_cache (ob_server_schema_service.cpp:4284) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=87] del sys_privs finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.497773] INFO [SHARE.SCHEMA] apply_sys_priv_schema_to_cache (ob_server_schema_service.cpp:4284) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=82] add sys_privs finish(schemas=[], ret=0) [2024-04-24 10:00:52.497908] INFO [SHARE.SCHEMA] apply_obj_priv_schema_to_cache (ob_server_schema_service.cpp:4285) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=63] del obj_privs finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.497973] INFO [SHARE.SCHEMA] apply_obj_priv_schema_to_cache (ob_server_schema_service.cpp:4285) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=58] add obj_privs finish(schemas=[], ret=0) [2024-04-24 10:00:52.498042] INFO [SHARE.SCHEMA] apply_dblink_schema_to_cache (ob_server_schema_service.cpp:4286) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=60] del dblinks finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.498131] INFO [SHARE.SCHEMA] apply_dblink_schema_to_cache (ob_server_schema_service.cpp:4286) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=77] add dblinks finish(schemas=[], ret=0) [2024-04-24 10:00:52.498211] INFO [SHARE.SCHEMA] apply_directory_schema_to_cache (ob_server_schema_service.cpp:4287) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=74] del directorys finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.498283] INFO [SHARE.SCHEMA] apply_directory_schema_to_cache (ob_server_schema_service.cpp:4287) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=60] add directorys finish(schemas=[], ret=0) [2024-04-24 10:00:52.498380] INFO [SHARE.SCHEMA] apply_context_schema_to_cache (ob_server_schema_service.cpp:4288) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=88] del contexts finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.498457] INFO [SHARE.SCHEMA] apply_context_schema_to_cache (ob_server_schema_service.cpp:4288) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=71] add contexts finish(schemas=[], ret=0) [2024-04-24 10:00:52.498541] INFO [SHARE.SCHEMA] apply_mock_fk_parent_table_schema_to_cache (ob_server_schema_service.cpp:4289) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=65] del mock_fk_parent_tables finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.498619] INFO [SHARE.SCHEMA] apply_mock_fk_parent_table_schema_to_cache (ob_server_schema_service.cpp:4289) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=71] add mock_fk_parent_tables finish(schemas=[], ret=0) [2024-04-24 10:00:52.498729] INFO [SHARE.SCHEMA] apply_rls_policy_schema_to_cache (ob_server_schema_service.cpp:4290) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=100] del rls_policys finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.498801] INFO [SHARE.SCHEMA] apply_rls_policy_schema_to_cache (ob_server_schema_service.cpp:4290) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=66] add rls_policys finish(schemas=[], ret=0) [2024-04-24 10:00:52.498944] INFO [SHARE.SCHEMA] apply_rls_group_schema_to_cache (ob_server_schema_service.cpp:4291) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=137] del rls_groups finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.499014] INFO [SHARE.SCHEMA] apply_rls_group_schema_to_cache (ob_server_schema_service.cpp:4291) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=64] add rls_groups finish(schemas=[], ret=0) [2024-04-24 10:00:52.499101] INFO [SHARE.SCHEMA] apply_rls_context_schema_to_cache (ob_server_schema_service.cpp:4292) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=75] del rls_contexts finish(schema_keys=[], ret=0) [2024-04-24 10:00:52.499177] INFO [SHARE.SCHEMA] apply_rls_context_schema_to_cache (ob_server_schema_service.cpp:4292) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=69] add rls_contexts finish(schemas=[], ret=0) [2024-04-24 10:00:52.499301] INFO [SHARE.SCHEMA] add_schema (ob_multi_version_schema_service.cpp:2199) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=58] add schema(tenant_id=1004, refreshed_schema_version=1713924046573584, new_schema_version=1713924050928872) [2024-04-24 10:00:52.499374] INFO [SHARE.SCHEMA] alloc_ (ob_schema_mem_mgr.cpp:115) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=60] alloc schema mgr(tmp_ptr=0x14b0474af980, tmp_ptr=0x14b0474af980) [2024-04-24 10:00:52.500626] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:749) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=70] assign tenant_infos_ cost(ret=0, ret="OB_SUCCESS", cost=22) [2024-04-24 10:00:52.500825] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:756) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=111] assign user_infos_ cost(ret=0, ret="OB_SUCCESS", cost=23) [2024-04-24 10:00:52.501043] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:757) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=144] assign database_infos_ cost(ret=0, ret="OB_SUCCESS", cost=29) [2024-04-24 10:00:52.501330] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=101][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:314138658}], scn_array=[{val:1713924052104362000, v:0}], curr_size=542, accum_size=3264, time_cost=391850) [2024-04-24 10:00:52.501346] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:758) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=239] assign database_name_map_ cost(ret=0, ret="OB_SUCCESS", cost=41) [2024-04-24 10:00:52.501638] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:759) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=203] assign tablegroup_infos_ cost(ret=0, ret="OB_SUCCESS", cost=53) [2024-04-24 10:00:52.501971] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:760) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=196] assign table_infos_ cost(ret=0, ret="OB_SUCCESS", cost=112) [2024-04-24 10:00:52.502185] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:761) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=153] assign index_infos_ cost(ret=0, ret="OB_SUCCESS", cost=31) [2024-04-24 10:00:52.502223] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [862319][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=102][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=505471, log_task={header:{begin_lsn:{lsn:314138658}, end_lsn:{lsn:314139200}, log_id:505471, min_scn:{val:1713924052104362000, v:0}, max_scn:{val:1713924052104362000, v:0}, data_len:486, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:314138658}, data_checksum:1663967765, accum_checksum:2613705461, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052109055, freeze_ts:1713924052109074, submit_ts:1713924052109269, flushed_ts:1713924052501773, gen_to_freeze cost time:19, gen_to_submit cost time:214, submit_to_flush cost time:392504}, fs_cb_begin_ts=1713924052502157, log_life_time=393102) [2024-04-24 10:00:52.502311] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:762) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=86] assign aux_vp_infos_ cost(ret=0, ret="OB_SUCCESS", cost=13) [2024-04-24 10:00:52.502427] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:763) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=73] assign lob_meta_infos_ cost(ret=0, ret="OB_SUCCESS", cost=23) [2024-04-24 10:00:52.502460] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=92][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052104362000, v:0}, log_ts:{val:1713924052104362000, v:0}, lsn:{lsn:314138714}, submit_ts:1713924052104362}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:568581}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=398072, ctx_lock_wait_time=6) [2024-04-24 10:00:52.502560] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:764) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=86] assign lob_piece_infos_ cost(ret=0, ret="OB_SUCCESS", cost=23) [2024-04-24 10:00:52.502772] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:765) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=184] assign drop_tenant_infos_ cost(ret=0, ret="OB_SUCCESS", cost=8) [2024-04-24 10:00:52.502994] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:766) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=141] assign table_id_map_ cost(ret=0, ret="OB_SUCCESS", cost=63) [2024-04-24 10:00:52.503207] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:767) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=123] assign table_name_map_ cost(ret=0, ret="OB_SUCCESS", cost=59) [2024-04-24 10:00:52.503367] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:768) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=105] assign index_name_map_ cost(ret=0, ret="OB_SUCCESS", cost=26) [2024-04-24 10:00:52.503448] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=96][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:1002200528}], scn_array=[{val:1713924052123200000, v:0}], curr_size=1573, accum_size=4840, time_cost=379317) [2024-04-24 10:00:52.503519] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:769) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=70] assign aux_vp_name_map_ cost(ret=0, ret="OB_SUCCESS", cost=65) [2024-04-24 10:00:52.503771] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:770) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=195] assign foreign_key_name_map_ cost(ret=0, ret="OB_SUCCESS", cost=37) [2024-04-24 10:00:52.503978] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:771) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=142] assign constraint_name_map_ cost(ret=0, ret="OB_SUCCESS", cost=32) [2024-04-24 10:00:52.504151] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=77][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=673430, log_task={header:{begin_lsn:{lsn:1002200528}, end_lsn:{lsn:1002202101}, log_id:673430, min_scn:{val:1713924052123200000, v:0}, max_scn:{val:1713924052123200000, v:0}, data_len:1517, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1002200528}, data_checksum:276248739, accum_checksum:4221427623, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052123735, freeze_ts:1713924052123753, submit_ts:1713924052123976, flushed_ts:1713924052503786, gen_to_freeze cost time:18, gen_to_submit cost time:241, submit_to_flush cost time:379810}, fs_cb_begin_ts=1713924052504020, log_life_time=380285) [2024-04-24 10:00:52.504175] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:772) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=128] assign hidden_table_name_map_ cost(ret=0, ret="OB_SUCCESS", cost=42) [2024-04-24 10:00:52.504240] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=574][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052123200000, v:0}, log_ts:{val:1713924052123200000, v:0}, lsn:{lsn:1002200584}, submit_ts:1713924052123200}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877475}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=381025, ctx_lock_wait_time=8) [2024-04-24 10:00:52.504672] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=161][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:79641312}], scn_array=[{val:1713924052130828000, v:0}], curr_size=424, accum_size=14187, time_cost=372961) [2024-04-24 10:00:52.504726] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=85][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:165107665}], scn_array=[{val:1713924052099241000, v:0}], curr_size=122, accum_size=1817, time_cost=241961) [2024-04-24 10:00:52.504816] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=123][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:298529908}], scn_array=[{val:1713924052029945001, v:0}], curr_size=122, accum_size=2580, time_cost=342521) [2024-04-24 10:00:52.504778] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=173][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67161219}], scn_array=[{val:1713924052080794000, v:0}], curr_size=122, accum_size=1010, time_cost=394072) [2024-04-24 10:00:52.504954] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=131][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:40362477}], scn_array=[{val:1713924052210944000, v:0}], curr_size=122, accum_size=693, time_cost=286125) [2024-04-24 10:00:52.505174] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=108][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179755457}], scn_array=[{val:1713924052127091000, v:0}], curr_size=538, accum_size=5249, time_cost=372075) [2024-04-24 10:00:52.505184] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=88][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=304871, log_task={header:{begin_lsn:{lsn:79641312}, end_lsn:{lsn:79641736}, log_id:304871, min_scn:{val:1713924052130828000, v:0}, max_scn:{val:1713924052130828000, v:0}, data_len:368, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:79641312}, data_checksum:1750356920, accum_checksum:374435105, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052131389, freeze_ts:1713924052131403, submit_ts:1713924052131546, flushed_ts:1713924052504909, gen_to_freeze cost time:14, gen_to_submit cost time:157, submit_to_flush cost time:373363}, fs_cb_begin_ts=1713924052505132, log_life_time=373743) [2024-04-24 10:00:52.505338] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [862141][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=89][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=259325, log_task={header:{begin_lsn:{lsn:67161219}, end_lsn:{lsn:67161341}, log_id:259325, min_scn:{val:1713924052080794000, v:0}, max_scn:{val:1713924052080794000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:67161219}, data_checksum:667309872, accum_checksum:3597124856, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052110350, freeze_ts:1713924052110364, submit_ts:1713924052110507, flushed_ts:1713924052505111, gen_to_freeze cost time:14, gen_to_submit cost time:157, submit_to_flush cost time:394604}, fs_cb_begin_ts=1713924052505295, log_life_time=394945) [2024-04-24 10:00:52.505391] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [874623][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=154][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=482564, log_task={header:{begin_lsn:{lsn:298529908}, end_lsn:{lsn:298530030}, log_id:482564, min_scn:{val:1713924052029945001, v:0}, max_scn:{val:1713924052029945001, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:298529908}, data_checksum:667309872, accum_checksum:2683124941, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052161855, freeze_ts:1713924052161934, submit_ts:1713924052162078, flushed_ts:1713924052505166, gen_to_freeze cost time:79, gen_to_submit cost time:223, submit_to_flush cost time:343088}, fs_cb_begin_ts=1713924052505331, log_life_time=343476) [2024-04-24 10:00:52.505573] INFO [SHARE.SCHEMA] assign (ob_schema_mgr.cpp:826) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=71] ObSchemaMgr assign cost(ret=0, ret="OB_SUCCESS", cost=4985) [2024-04-24 10:00:52.505656] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2BFC8F2C2-0-0] [lt=126][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052130828000, v:0}, log_ts:{val:1713924052130828000, v:0}, lsn:{lsn:79641368}, submit_ts:1713924052130828}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176928}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=374806, ctx_lock_wait_time=14) [2024-04-24 10:00:52.505718] INFO [SHARE.SCHEMA] put (ob_schema_mgr_cache.cpp:502) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=120] put schema mgr(schema version=1713924050928872) [2024-04-24 10:00:52.505811] INFO [SHARE.SCHEMA] put (ob_schema_mgr_cache.cpp:576) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=71] dst schema mgr item ptr(tenant_id=1004, dst_item=0x14b11fbbe930, dst_timestamp=1713924052505211, dst_schema_version=1713924050928872, target_pos=9) [2024-04-24 10:00:52.505932] INFO [SHARE.SCHEMA] alloc_and_put_schema_mgr_ (ob_multi_version_schema_service.cpp:2281) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=114] put schema mgr succeed(schema_version=1713924050928872, eliminated_schema_version=-1, tenant_id=1004) [2024-04-24 10:00:52.506010] INFO [SHARE.SCHEMA] update_refreshed_version (ob_schema_store.cpp:60) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=69] [SCHEMA_STORE] schema store update version(tenant_id=1004, version=1713924050928872, refreshed_version=1713924050928872, received_version=1713924050928872) [2024-04-24 10:00:52.506125] INFO [SHARE.SCHEMA] add_schema (ob_multi_version_schema_service.cpp:2234) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=106] [REFRESH_SCHEMA] change refreshed_schema_version with new mode(tenant_id=1004, new_schema_version=1713924050928872) [2024-04-24 10:00:52.506357] INFO [SHARE.SCHEMA] try_gc_allocator_when_add_schema_ (ob_multi_version_schema_service.cpp:3865) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=116] try to gc allocator's schema mgr which schema version is less than reserve_version(tenant_id=1004, eli_schema_version=1713924034253440, local_version=1713924050928872, reserve_version=1713924034374984) [2024-04-24 10:00:52.506850] INFO [SHARE.SCHEMA] free_schema_mgr (ob_schema_mem_mgr.cpp:260) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=129] [SCHEMA_RELEASE] free schema mgr(tenant_id=1004, schema_version=1713924034253440, timestamp_in_slot=1713924035663260) [2024-04-24 10:00:52.506987] INFO [SHARE.SCHEMA] try_gc_allocator_when_add_schema_ (ob_multi_version_schema_service.cpp:3884) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=116] finish gc allocator's schema mgr when add schema(ret=0, ret="OB_SUCCESS", tenant_id=1004, total_schema_ptr_cnt=12, remain_schema_ptr_cnt=11, reserve_version=1713924034374984, cost_ts=725) [2024-04-24 10:00:52.507102] INFO [SHARE.SCHEMA] add_schema (ob_multi_version_schema_service.cpp:2241) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=89] finish add schema(ret=0, ret="OB_SUCCESS", tenant_id=1004, new_schema_version=1713924050928872, cost_ts=-7811) [2024-04-24 10:00:52.507196] INFO [SHARE.SCHEMA] refresh_increment_schema (ob_server_schema_service.cpp:5856) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=85] change schema version(schema_status={tenant_id:1004, snapshot_timestamp:-1, readable_schema_version:-1}, schema_version=1713924050928872, core_schema_version=1713877451559520) [2024-04-24 10:00:52.507335] INFO [SHARE.SCHEMA] refresh_schema (ob_server_schema_service.cpp:5434) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=124] [REFRESH_SCHEMA] finish refresh increment schema(ret=0, schema_status={tenant_id:1004, snapshot_timestamp:-1, readable_schema_version:-1}, current schema_version=1713924050928872, cost=196491) [2024-04-24 10:00:52.507419] INFO [SHARE.SCHEMA] set_tenant_received_broadcast_version (ob_multi_version_schema_service.cpp:4100) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=64] try to set tenant received_broadcast_version(ret=0, tenant_id=1004, version=1713924050928872) [2024-04-24 10:00:52.507502] INFO [SHARE.SCHEMA] refresh_tenant_schema (ob_multi_version_schema_service.cpp:2889) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=68] [REFRESH_SCHEMA] end refresh and add schema by tenant(ret=0, ret="OB_SUCCESS", tenant_id=1004, cost=198240) [2024-04-24 10:00:52.507574] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2572) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=59] [REFRESH_SCHEMA] end refresh and add schema(ret=0, ret="OB_SUCCESS", tenant_ids=[1004], cost=198391) [2024-04-24 10:00:52.507646] INFO [SERVER] process_async_refresh_tasks (ob_server_schema_updater.cpp:405) [861416][SerScheQueue1][T0][YB420A000003-000616C2AA6924EF-0-0] [lt=65] try to async refresh schema(ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:52.511011] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=71] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.511172] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=140] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.511258] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=75] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.512344] INFO [SHARE.SCHEMA] set_tenant_received_broadcast_version (ob_multi_version_schema_service.cpp:4100) [861898][T1_L0_G0][T1][YB420A000002-000616C2AD7C034E-0-0] [lt=90] try to set tenant received_broadcast_version(ret=0, tenant_id=1004, version=1713924050928872) [2024-04-24 10:00:52.512436] INFO [SERVER] switch_schema (ob_service.cpp:1383) [861898][T1_L0_G0][T1][YB420A000002-000616C2AD7C034E-0-0] [lt=86] switch schema(ret=0, schema_info={schema_version:1713924050928872, tenant_id:1004, sequence_id:5497558139159}) [2024-04-24 10:00:52.513959] INFO [SHARE.SCHEMA] update_consensus_version (ob_schema_store.cpp:96) [861898][T1_L0_G0][T1][YB420A000002-000616C2AD7C034E-0-0] [lt=99] [SCHEMA_STORE] schema store update version(tenant_id=1004, version=1713924050928872, consensus_version=1713924050928872) [2024-04-24 10:00:52.514076] INFO [SHARE.SCHEMA] set_tenant_broadcast_consensus_version (ob_multi_version_schema_service.cpp:4079) [861898][T1_L0_G0][T1][YB420A000002-000616C2AD7C034E-0-0] [lt=86] try to set tenant broadcast consensus version(ret=0, ret="OB_SUCCESS", tenant_id=1004, consensus_version=1713924050928872) [2024-04-24 10:00:52.520450] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [862171][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=84] ====== tenant freeze timer task ====== [2024-04-24 10:00:52.520543] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [862171][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=56] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=75497472, ctx.total_memstore_used_=109051904, ctx.total_memstore_hold_=109051904, memstore_freeze_trigger_limit_=91749580, tenant_id=1002) [2024-04-24 10:00:52.520651] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [862171][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=75] [TenantFreezer] oldest frozen memtable(list="") [2024-04-24 10:00:52.523730] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:213) [862246][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=114] start to check and clear obj lock(ls->get_ls_id()={id:1}) [2024-04-24 10:00:52.524089] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:213) [862246][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=76] start to check and clear obj lock(ls->get_ls_id()={id:1001}) [2024-04-24 10:00:52.524309] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [861898][T1_L0_G0][T1][YB420A000002-000616C2AA6C607D-0-0] [lt=62] [RS_MGR] force set rs list(master_rs="10.0.0.2:2882") [2024-04-24 10:00:52.524397] INFO [SERVER] handle_heartbeat (ob_service.cpp:3094) [861898][T1_L0_G0][T1][YB420A000002-000616C2AA6C607D-0-0] [lt=68] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"10.0.0.3:2882", server_id:1, rs_addr:"10.0.0.2:2882", rs_server_status:1, epoch_id:5}, hb_response={zone:"zone1", server:"10.0.0.3:2882", sql_port:2881, build_version:"4.2.2.0_100010012024022719-c984fe7cb7a4cef85a40323a0d073f0c9b7b8235(Feb 27 2024 19:21:00)", start_service_time:1713875297913711, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=86) [2024-04-24 10:00:52.524523] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:213) [862246][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=94] start to check and clear obj lock(ls->get_ls_id()={id:1002}) [2024-04-24 10:00:52.527105] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:834) [874425][T1004_BalanceEx][T1004][YB420A000003-000616C2BB69A61A-0-0] [lt=93] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2024-04-24 10:00:52.527211] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:171) [874425][T1004_BalanceEx][T1004][YB420A000003-000616C2BB69A61A-0-0] [lt=95] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2024-04-24 10:00:52.529978] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=114] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8905367552, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:52.532306] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=111] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=31457, clean_num=31457) [2024-04-24 10:00:52.532995] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2323545][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0535-0-0] [lt=103][errcode=-4023] tx is 2pc logging(this={this:0x14b14807aed0, ref:3, trans_id:{txid:176929}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925052083398, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052225549], ctx_create_time:1713924052213383}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052221452, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052225180000, v:0}, log_ts:{val:1713924052225180000, v:0}, lsn:{lsn:79641792}, submit_ts:1713924052225180}, this:0x14b14807de40, is_inited_:true, trans_id:{txid:176929}, ls_id:{id:1}, ctx:0x14b14807aed0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176929}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924052225180000, v:0}, start_scn:{val:1713924052225180000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877454130248 max_table_version=1713877454130248 trans_version={val:1713924052225180000, v:0} commit_version={val:1713924052225180000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176929} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3248666907 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052225180000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052225180, block_frozen_memtable:null}) [2024-04-24 10:00:52.533165] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2323545][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0535-0-0] [lt=164][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b14807aed0, ref:3, trans_id:{txid:176929}, tenant_id:1004, is_exiting:false, trans_expired_time:1713925052083398, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052225549], ctx_create_time:1713924052213383}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052221452, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052225180000, v:0}, log_ts:{val:1713924052225180000, v:0}, lsn:{lsn:79641792}, submit_ts:1713924052225180}, this:0x14b14807de40, is_inited_:true, trans_id:{txid:176929}, ls_id:{id:1}, ctx:0x14b14807aed0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176929}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924052225180000, v:0}, start_scn:{val:1713924052225180000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877454130248 max_table_version=1713877454130248 trans_version={val:1713924052225180000, v:0} commit_version={val:1713924052225180000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176929} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3248666907 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052225180000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052225180, block_frozen_memtable:null}) [2024-04-24 10:00:52.533307] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2323545][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0535-0-0] [lt=130][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176929}) [2024-04-24 10:00:52.533407] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2323545][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0535-0-0] [lt=89][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176929}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924052528477, cluster_id:1713874979}, expire_ts:1713925052083381, commit_start_scn:{val:1713924052215814000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:52.533541] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2323545][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0535-0-0] [lt=111][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176929}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:5, timestamp:1713924052528477, cluster_id:1713874979}, expire_ts:1713925052083381, commit_start_scn:{val:1713924052215814000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:52.536737] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=83][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.537519] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=111] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.550467] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [874589][T1004_L2_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=230] current clock is (clock_=0, max_seq_=0, mem_limit=3181127926, cur_mem_hold=297795584, attr_.tenant_id_=1004) [2024-04-24 10:00:52.553577] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=79] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.558950] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [862229][T1002_TenantMet][T1002][Y0-0000000000000000-0-0] [lt=110] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1850, free_obj_hold_cnt:214, allocator used:8206464, allocator total:8437632}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:215, free_obj_hold_cnt:39, allocator used:16660114, allocator total:16660114}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():111104, full allocator used:1600, full allocator total:176640}, tablets_mem=25274386, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:84, free_obj_hold_cnt:1045, allocator used:2239936, allocator total:2354688}, pending_cnt=0, wait_gc_count=0, tablet count=2065) [2024-04-24 10:00:52.561527] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [862262][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=64] ====== traversal_flush timer task ====== [2024-04-24 10:00:52.562024] INFO [MDS] flush (mds_table_mgr.cpp:195) [862262][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=77] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60379}, rec_scn:{val:1713895310268028001, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1713895310268028003, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1713895310525506001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1713895310525506002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1713895310678201001, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1713895310678201002, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1713895310852482001, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1713895310852482002, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1713895310852482003, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1713895310852482004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1713895310992911001, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1713895310992911002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1713895311111630001, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1713895311260927001, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1713895311260927002, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1713895311260927003, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1713895311398568001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1713895311588474001, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1713895311588474003, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1713895311588474004, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1713895311783317001, v:0}}, {tablet_id:{id:378}, rec_scn:{val:1713895311783317002, v:0}}, {tablet_id:{id:101036}, rec_scn:{val:1713895311783317003, v:0}}, {tablet_id:{id:60005}, rec_scn:{val:1713895311783317004, v:0}}, {tablet_id:{id:60130}, rec_scn:{val:1713895311984459001, v:0}}, {tablet_id:{id:101064}, rec_scn:{val:1713895311984459002, v:0}}, {tablet_id:{id:101080}, rec_scn:{val:1713895311984459003, v:0}}, {tablet_id:{id:50424}, rec_scn:{val:1713895311984459004, v:0}}, {tablet_id:{id:197}, rec_scn:{val:1713895311984459005, v:0}}, {tablet_id:{id:50381}, rec_scn:{val:1713895312133482001, v:0}}, {tablet_id:{id:103}, rec_scn:{val:1713895312133482002, v:0}}, {tablet_id:{id:50373}, rec_scn:{val:1713895312274829001, v:0}}, }, max_consequent_callbacked_scn={val:1713924051925376000, v:0}, *this={this:0x14b147070510, is_inited:true, freezing_scn:{val:1713877162774754000, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895310268028001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924045940260000, v:0}, tablet_change_checkpoint_scn:{val:1713875507773116000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924066491232000, latest_log_ts:{val:1713924049917601001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713890395635598001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:5, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:52.562581] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=86] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.562556] INFO [MDS] flush (mds_table_mgr.cpp:195) [862262][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=220] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:200589}, rec_scn:{val:1713895385940970001, v:0}}, {tablet_id:{id:201365}, rec_scn:{val:1713895386138343001, v:0}}, {tablet_id:{id:200929}, rec_scn:{val:1713895386138343002, v:0}}, {tablet_id:{id:200529}, rec_scn:{val:1713895386246298001, v:0}}, {tablet_id:{id:200197}, rec_scn:{val:1713895386246298003, v:0}}, {tablet_id:{id:200127}, rec_scn:{val:1713895386443363001, v:0}}, {tablet_id:{id:200251}, rec_scn:{val:1713895386443363003, v:0}}, {tablet_id:{id:201051}, rec_scn:{val:1713895386883002001, v:0}}, {tablet_id:{id:200767}, rec_scn:{val:1713895386883002002, v:0}}, {tablet_id:{id:201463}, rec_scn:{val:1713895387425291001, v:0}}, {tablet_id:{id:200999}, rec_scn:{val:1713895387572813001, v:0}}, {tablet_id:{id:201235}, rec_scn:{val:1713895388449421002, v:0}}, {tablet_id:{id:201309}, rec_scn:{val:1713895388449421004, v:0}}, {tablet_id:{id:200325}, rec_scn:{val:1713895388659263001, v:0}}, {tablet_id:{id:201025}, rec_scn:{val:1713895388717594001, v:0}}, {tablet_id:{id:200971}, rec_scn:{val:1713895388717594003, v:0}}, {tablet_id:{id:201081}, rec_scn:{val:1713895388717594005, v:0}}, {tablet_id:{id:200347}, rec_scn:{val:1713895389791968002, v:0}}, {tablet_id:{id:201151}, rec_scn:{val:1713895389992852001, v:0}}, {tablet_id:{id:200913}, rec_scn:{val:1713895389992852002, v:0}}, {tablet_id:{id:201483}, rec_scn:{val:1713895390122768001, v:0}}, {tablet_id:{id:201351}, rec_scn:{val:1713895390122768002, v:0}}, {tablet_id:{id:201205}, rec_scn:{val:1713895390122768004, v:0}}, {tablet_id:{id:201335}, rec_scn:{val:1713895390348851001, v:0}}, {tablet_id:{id:200911}, rec_scn:{val:1713895390455961001, v:0}}, {tablet_id:{id:200193}, rec_scn:{val:1713895390455961003, v:0}}, {tablet_id:{id:201239}, rec_scn:{val:1713895390714357007, v:0}}, {tablet_id:{id:200301}, rec_scn:{val:1713895391010940001, v:0}}, {tablet_id:{id:200703}, rec_scn:{val:1713895391146241001, v:0}}, {tablet_id:{id:200379}, rec_scn:{val:1713895391146241002, v:0}}, {tablet_id:{id:201375}, rec_scn:{val:1713895391240940001, v:0}}, {tablet_id:{id:201449}, rec_scn:{val:1713895391319921001, v:0}}, }, max_consequent_callbacked_scn={val:1713924051925376000, v:0}, *this={this:0x14b13438c510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1713895385940970001, v:0}, clog_base_lsn:{lsn:335523840}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924046174250000, v:0}, tablet_change_checkpoint_scn:{val:1713890381649962000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:7, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:52.563090] INFO [MDS] flush (mds_table_mgr.cpp:195) [862262][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=231] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:200876}, rec_scn:{val:1713895298278308001, v:0}}, {tablet_id:{id:200048}, rec_scn:{val:1713895298438136001, v:0}}, {tablet_id:{id:201384}, rec_scn:{val:1713895298438136002, v:0}}, {tablet_id:{id:200608}, rec_scn:{val:1713895298550042001, v:0}}, {tablet_id:{id:200450}, rec_scn:{val:1713895298611638001, v:0}}, {tablet_id:{id:200810}, rec_scn:{val:1713895298611638002, v:0}}, {tablet_id:{id:201394}, rec_scn:{val:1713895298744317001, v:0}}, {tablet_id:{id:201250}, rec_scn:{val:1713895298744317002, v:0}}, {tablet_id:{id:201318}, rec_scn:{val:1713895298744317003, v:0}}, {tablet_id:{id:200822}, rec_scn:{val:1713895298900066001, v:0}}, {tablet_id:{id:200874}, rec_scn:{val:1713895298900066002, v:0}}, {tablet_id:{id:200880}, rec_scn:{val:1713895299047127001, v:0}}, {tablet_id:{id:201270}, rec_scn:{val:1713895299149533001, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1713895299149533002, v:0}}, {tablet_id:{id:201434}, rec_scn:{val:1713895299252327001, v:0}}, {tablet_id:{id:201126}, rec_scn:{val:1713895299252327002, v:0}}, {tablet_id:{id:200124}, rec_scn:{val:1713895299252327003, v:0}}, {tablet_id:{id:200976}, rec_scn:{val:1713895299252327004, v:0}}, {tablet_id:{id:200566}, rec_scn:{val:1713895299446962001, v:0}}, {tablet_id:{id:201450}, rec_scn:{val:1713895299446962003, v:0}}, {tablet_id:{id:200312}, rec_scn:{val:1713895299446962004, v:0}}, {tablet_id:{id:200690}, rec_scn:{val:1713895299446962005, v:0}}, {tablet_id:{id:200808}, rec_scn:{val:1713895299654655001, v:0}}, {tablet_id:{id:200130}, rec_scn:{val:1713895299654655002, v:0}}, {tablet_id:{id:201138}, rec_scn:{val:1713895299654655003, v:0}}, {tablet_id:{id:200314}, rec_scn:{val:1713895299654655004, v:0}}, {tablet_id:{id:200031}, rec_scn:{val:1713895299654655005, v:0}}, {tablet_id:{id:200662}, rec_scn:{val:1713895299850465001, v:0}}, {tablet_id:{id:200492}, rec_scn:{val:1713895299955311001, v:0}}, {tablet_id:{id:200678}, rec_scn:{val:1713895299955311002, v:0}}, {tablet_id:{id:200332}, rec_scn:{val:1713895300142160001, v:0}}, {tablet_id:{id:201114}, rec_scn:{val:1713895300142160002, v:0}}, }, max_consequent_callbacked_scn={val:1713924052017398000, v:0}, *this={this:0x14b133736510, is_inited:true, freezing_scn:{val:1713890363233517001, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_create_status:1, clog_checkpoint_scn:{val:1713895298278308001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1713924046174250000, v:0}, tablet_change_checkpoint_scn:{val:1713890384892147000, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:2, proposal_id:0, palf_env_:0x14b1679f2030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:1942234752, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:52.563312] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [862262][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=189] succeed to traversal_flush(ret=0, ls_cnt=3) [2024-04-24 10:00:52.564955] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [862259][T1002_ReqMemEvi][T1002][Y0-0000000000000000-0-0] [lt=127] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=16106127, evict_high_size_level=90000, evict_batch_count=0, elapse_time=10, size_used=6710, mem_used=12478464) [2024-04-24 10:00:52.568015] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=303][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:88132621}], scn_array=[{val:1713924052130828000, v:0}], curr_size=122, accum_size=3431, time_cost=384858) [2024-04-24 10:00:52.571097] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=77][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179755995}], scn_array=[{val:1713924052284612000, v:0}], curr_size=122, accum_size=5371, time_cost=65711) [2024-04-24 10:00:52.571267] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=82][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.572040] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD2AD-0-0] [lt=77][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=299923, log_task={header:{begin_lsn:{lsn:40362477}, end_lsn:{lsn:40362599}, log_id:299923, min_scn:{val:1713924052210944000, v:0}, max_scn:{val:1713924052210944000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:40362028}, prev_proposal_id:5, committed_end_lsn:{lsn:40362477}, data_checksum:1715074492, accum_checksum:44674791, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924052218617, freeze_ts:1713924052218624, submit_ts:1713924052218682, flushed_ts:1713924052505364, gen_to_freeze cost time:7, gen_to_submit cost time:65, submit_to_flush cost time:286682}, fs_cb_begin_ts=1713924052571993, log_life_time=353376) [2024-04-24 10:00:52.572079] INFO [LIB] stat (utility.h:1197) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD1422-0-0] [lt=66] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=35, this=0x14b186b476a0, extra_info="palf_id:1") [2024-04-24 10:00:52.572232] INFO [LIB] stat (utility.h:1197) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD1422-0-0] [lt=139] [PALF STAT LOG LIFE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=241829, this=0x14b186b478d0, extra_info="palf_id:1") [2024-04-24 10:00:52.572249] WDIAG [STORAGE.TRANS] fill_redo_log (ob_memtable_context.cpp:655) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=81][errcode=-5199] fill_redo_log fail(ret=-5199, trans_id="{txid:176931}", buf="muta›gƒ3", buf_len=1964, buf_pos=0, pending_log_size=75, *this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465782000 trans_version={val:1713924052568961000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176931} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=1 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}) [2024-04-24 10:00:52.572343] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD1422-0-0] [lt=103] [PALF STAT LOG TASK TIME](palf_id=1, self="10.0.0.3:2882", total_slide_log_cnt=6, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=133, avg_log_submit_to_flush_time=222002, avg_log_submit_to_slide_time=241696) [2024-04-24 10:00:52.572374] WDIAG [STORAGE.TRANS] fill_redo_log_ (ob_trans_part_ctx.cpp:2673) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=118][errcode=-5199] fill redo log failed(ret=-5199, ret="OB_ERR_TOO_BIG_ROWSIZE", *this={this:0x14b148003290, ref:3, trans_id:{txid:176931}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924082541480, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052572154], ctx_create_time:1713924052549745}{ls_id:{id:1}, session_id:1, part_trans_action:2, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79642727}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052548472, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:0}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176931}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176931}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924052568961000, v:0}, start_scn:{val:1713924052225180002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465782000 trans_version={val:1713924052568961000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176931} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=1 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052225180002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052567410, block_frozen_memtable:null}) [2024-04-24 10:00:52.572427] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD1422-0-0] [lt=76][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=364251, log_task={header:{begin_lsn:{lsn:179755457}, end_lsn:{lsn:179755995}, log_id:364251, min_scn:{val:1713924052127091000, v:0}, max_scn:{val:1713924052127091000, v:0}, data_len:482, proposal_id:5, prev_lsn:{lsn:179754433}, prev_proposal_id:5, committed_end_lsn:{lsn:179755457}, data_checksum:811187835, accum_checksum:3397590830, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924052132408, freeze_ts:1713924052132419, submit_ts:1713924052132586, flushed_ts:1713924052505391, gen_to_freeze cost time:11, gen_to_submit cost time:178, submit_to_flush cost time:372805}, fs_cb_begin_ts=1713924052572003, log_life_time=439595) [2024-04-24 10:00:52.572518] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C5A-0-0] [lt=131][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=301043, log_task={header:{begin_lsn:{lsn:165107665}, end_lsn:{lsn:165107787}, log_id:301043, min_scn:{val:1713924052099241000, v:0}, max_scn:{val:1713924052099241000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:165107543}, prev_proposal_id:5, committed_end_lsn:{lsn:165107665}, data_checksum:1715074492, accum_checksum:3289229797, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924052262518, freeze_ts:1713924052262526, submit_ts:1713924052262590, flushed_ts:1713924052504978, gen_to_freeze cost time:8, gen_to_submit cost time:72, submit_to_flush cost time:242388}, fs_cb_begin_ts=1713924052572468, log_life_time=309950) [2024-04-24 10:00:52.572511] WDIAG [STORAGE.TRANS] submit_redo_log_ (ob_trans_part_ctx.cpp:2765) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=129][errcode=-5199] encounter too big row size error, need extend log buffer(ret=-5199, *this={this:0x14b148003290, ref:3, trans_id:{txid:176931}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924082541480, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052572154], ctx_create_time:1713924052549745}{ls_id:{id:1}, session_id:1, part_trans_action:2, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79642727}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052548472, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:0}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176931}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176931}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924052568961000, v:0}, start_scn:{val:1713924052225180002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465782000 trans_version={val:1713924052568961000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176931} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=1 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052225180002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052567410, block_frozen_memtable:null}) [2024-04-24 10:00:52.572639] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=92][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052129649000, v:0}, log_ts:{val:1713924052129649000, v:0}, lsn:{lsn:314139256}, submit_ts:1713924052129649}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568582}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=442967, ctx_lock_wait_time=14) [2024-04-24 10:00:52.572654] INFO [STORAGE.TRANS] submit_redo_log_ (ob_trans_part_ctx.cpp:2779) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=113] extend log buffer success(*this={this:0x14b148003290, ref:3, trans_id:{txid:176931}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924082541480, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052572154], ctx_create_time:1713924052549745}{ls_id:{id:1}, session_id:1, part_trans_action:2, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79642727}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052548472, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:0}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:18446744073709551615, v:3}, log_ts:{val:18446744073709551615, v:3}, lsn:{lsn:18446744073709551615}, submit_ts:0}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176931}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176931}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924052568961000, v:0}, start_scn:{val:1713924052225180002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465782000 trans_version={val:1713924052568961000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176931} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=1 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052225180002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052567410, block_frozen_memtable:null}, log_block={fill_buf_:{buf:0x14b045804030, default_buf:0x14b0fcc55178, len:1965056}, replay_buf_:null, len_:1965056, pos_:54, cur_log_type_:0, cb_arg_array_:[], big_segment_buf_:NULL}) [2024-04-24 10:00:52.572769] INFO [PALF] read (log_cache.cpp:100) [861709][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=100] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=7343, hit_cnt=5, read_cnt=5, hit rate=1.000000000000000000e+00) [2024-04-24 10:00:52.573001] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=151][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67161341}], scn_array=[{val:1713924052118103000, v:0}], curr_size=498, accum_size=1508, time_cost=66799) [2024-04-24 10:00:52.572986] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=186][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:1002202101}], scn_array=[{val:1713924052123200001, v:0}], curr_size=122, accum_size=4962, time_cost=68617) [2024-04-24 10:00:52.572986] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=94][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:79641736}], scn_array=[{val:1713924052225180000, v:0}], curr_size=935, accum_size=15122, time_cost=67580) [2024-04-24 10:00:52.573341] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=113][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:298530030}], scn_array=[{val:1713924052199925000, v:0}], curr_size=1028, accum_size=3608, time_cost=67224) [2024-04-24 10:00:52.573975] INFO [LIB] stat (utility.h:1197) [861712][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=95] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=289, this=0x14b1976048f8, extra_info="") [2024-04-24 10:00:52.574183] INFO [LIB] stat (utility.h:1197) [861712][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=175] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=687, this=0x14b197604b28, extra_info="") [2024-04-24 10:00:52.574327] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=88][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052199925000, v:0}, log_ts:{val:1713924052199925000, v:0}, lsn:{lsn:298530086}, submit_ts:1713924052201452}, this:0x14b0fbee5680, is_inited_:true, trans_id:{txid:548742}, ls_id:{id:1}, ctx:0x14b0fbee2710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=372855, ctx_lock_wait_time=8) [2024-04-24 10:00:52.574718] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=195][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052298485000, v:0}, log_ts:{val:1713924052298485000, v:0}, lsn:{lsn:298530572}, submit_ts:1713924052298485}, this:0x14b0fc124480, is_inited_:true, trans_id:{txid:548743}, ls_id:{id:1}, ctx:0x14b0fc121510, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=276213, ctx_lock_wait_time=10) [2024-04-24 10:00:52.574795] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=258][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052118103000, v:0}, log_ts:{val:1713924052118103000, v:0}, lsn:{lsn:67161397}, submit_ts:1713924052118103}, this:0x14b168f7bb00, is_inited_:true, trans_id:{txid:869521}, ls_id:{id:1}, ctx:0x14b168f78b90, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=456672, ctx_lock_wait_time=6) [2024-04-24 10:00:52.576310] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [862430][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=165] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=90000, evict_batch_count=0, elapse_time=12, size_used=8596, mem_used=14558208) [2024-04-24 10:00:52.577731] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:488) [861547][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=668] worker thread began to shrink(tenant_->id()=1004, group_id_=19, token=0) [2024-04-24 10:00:52.581053] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [861695][T1_ArcSender][T1][YB420A000003-000616C2A8B8F0E7-0-0] [lt=98][errcode=-4018] try free send task failed(ret=-4018) [2024-04-24 10:00:52.581491] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [862440][T1001_ArcSender][T1001][YB420A000003-000616C2AF28F0E7-0-0] [lt=58][errcode=-4018] try free send task failed(ret=-4018) [2024-04-24 10:00:52.583605] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=110] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.583748] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=123] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.583808] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:503) [2325129][T1004_L0_G19][T1004][Y0-0000000000000000-0-0] [lt=98] worker thread exit(tenant_->id()=1004, workers_.get_size()=1) [2024-04-24 10:00:52.583912] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=154] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.583970] INFO [SHARE] end_run (ob_tenant_base.cpp:332) [2325129][T1004_L0_G19][T0][Y0-0000000000000000-0-0] [lt=132] tenant thread end_run(id_=1004, ret=0, thread_count_=237) [2024-04-24 10:00:52.584051] INFO unregister_pm (ob_page_manager.cpp:50) [2325129][T1004_L0_G19][T0][Y0-0000000000000000-0-0] [lt=73] unregister pm finish(&pm=0x14b064952d40, pm.get_tid()=2325129) [2024-04-24 10:00:52.590166] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [861688][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=99] ====== traversal_flush timer task ====== [2024-04-24 10:00:52.590556] INFO [MDS] flush (mds_table_mgr.cpp:195) [861688][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=92] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1713895261489897001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1713895262106069001, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1713895262106069002, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1713895262608522001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1713895262608522002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1713895263027701001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1713895263027701002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1713895263027701003, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1713895263109231001, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1713895263201088001, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1713895263226856002, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1713895263471989001, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1713895263515510001, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1713895263616210002, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1713895263616210003, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1713895263734257001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1713895263795311001, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1713895263838851001, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1713895263891959001, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1713895263936540001, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1713895263945526001, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1713895263990416001, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1713895264026604001, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1713895264026604002, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1713895264026604003, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1713895264026604004, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1713895264138915001, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1713895264138915002, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1713895264138915003, v:0}}, {tablet_id:{id:60342}, rec_scn:{val:1713895264138915004, v:0}}, {tablet_id:{id:378}, rec_scn:{val:1713895264138915005, v:0}}, {tablet_id:{id:101036}, rec_scn:{val:1713895264138915006, v:0}}, }, max_consequent_callbacked_scn={val:1713924052127091000, v:0}, *this={this:0x14b186b16510, is_inited:true, freezing_scn:{val:1713895202277178000, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895204295552002, v:0}, clog_base_lsn:{lsn:67104768}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1713875366171467000, v:0}, all_id_meta:{id_meta:[{limited_id:1713924068636985000, latest_log_ts:{val:1713924051990986001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713918035834957001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1713903554859671002, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:2, proposal_id:0, palf_env_:0x14b1945f8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:52.590740] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [861688][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=156] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-04-24 10:00:52.601163] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [861744][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=92] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1713924050928872, tenant_id:1004, sequence_id:5497558139159}) [2024-04-24 10:00:52.601263] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [861744][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=92] try reload schema success(schema_version=1713878795542168, refresh_schema_info={schema_version:1713924050928872, tenant_id:1004, sequence_id:5497558139159}, schema_ret=0) [2024-04-24 10:00:52.601329] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=100] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1713924050928872, tenant_id:1004, sequence_id:5497558139159}) [2024-04-24 10:00:52.601431] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:299) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=68] [REFRESH_SCHEMA] begin refresh schema, (begin_time=1713924052601412) [2024-04-24 10:00:52.601555] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2465) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=116] [REFRESH_SCHEMA] start to refresh and add schema(tenant_ids=[]) [2024-04-24 10:00:52.601754] INFO [SHARE.SCHEMA] refresh_tenant_schema (ob_multi_version_schema_service.cpp:2795) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=157] [REFRESH_SCHEMA] start to refresh and add schema by tenant(tenant_id=1) [2024-04-24 10:00:52.604539] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=85] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1001, min_version={val:1713924052035048000, v:0}, base_version={val:1713924051825760000, v:0}, server_count=2, svr_array=[{addr:"10.0.0.3:2882", version:{val:1713924051825760000, v:0}, valid_part_count:0, total_part_count:1, generate_tstamp:1713924052604394, is_skipped:true}, {addr:"10.0.0.2:2882", version:{val:1713924052035048000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1713924052524290, is_skipped:false}], skip_server_count=1, skip_servers=["10.0.0.3:2882"]) [2024-04-24 10:00:52.606178] INFO [SHARE.SCHEMA] set_tenant_received_broadcast_version (ob_multi_version_schema_service.cpp:4100) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=89] try to set tenant received_broadcast_version(ret=0, tenant_id=1, version=1713878795542168) [2024-04-24 10:00:52.606307] INFO [SHARE.SCHEMA] refresh_tenant_schema (ob_multi_version_schema_service.cpp:2889) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=112] [REFRESH_SCHEMA] end refresh and add schema by tenant(ret=0, ret="OB_SUCCESS", tenant_id=1, cost=4448) [2024-04-24 10:00:52.606428] INFO [SHARE.SCHEMA] refresh_tenant_schema (ob_multi_version_schema_service.cpp:2795) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=71] [REFRESH_SCHEMA] start to refresh and add schema by tenant(tenant_id=1001) [2024-04-24 10:00:52.608263] INFO [SHARE.SCHEMA] set_tenant_received_broadcast_version (ob_multi_version_schema_service.cpp:4100) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=69] try to set tenant received_broadcast_version(ret=0, tenant_id=1001, version=1713877131234192) [2024-04-24 10:00:52.608353] INFO [SHARE.SCHEMA] refresh_tenant_schema (ob_multi_version_schema_service.cpp:2889) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=76] [REFRESH_SCHEMA] end refresh and add schema by tenant(ret=0, ret="OB_SUCCESS", tenant_id=1001, cost=1841) [2024-04-24 10:00:52.608422] INFO [SHARE.SCHEMA] refresh_tenant_schema (ob_multi_version_schema_service.cpp:2795) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=63] [REFRESH_SCHEMA] start to refresh and add schema by tenant(tenant_id=1002) [2024-04-24 10:00:52.610157] INFO [SHARE.SCHEMA] set_tenant_received_broadcast_version (ob_multi_version_schema_service.cpp:4100) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=82] try to set tenant received_broadcast_version(ret=0, tenant_id=1002, version=1713890377976304) [2024-04-24 10:00:52.610293] INFO [SHARE.SCHEMA] refresh_tenant_schema (ob_multi_version_schema_service.cpp:2889) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=112] [REFRESH_SCHEMA] end refresh and add schema by tenant(ret=0, ret="OB_SUCCESS", tenant_id=1002, cost=1775) [2024-04-24 10:00:52.610403] INFO [SHARE.SCHEMA] refresh_tenant_schema (ob_multi_version_schema_service.cpp:2795) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=101] [REFRESH_SCHEMA] start to refresh and add schema by tenant(tenant_id=1003) [2024-04-24 10:00:52.611230] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=96] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.611509] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=82] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.611620] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=89] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.612363] INFO [SHARE.SCHEMA] set_tenant_received_broadcast_version (ob_multi_version_schema_service.cpp:4100) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=130] try to set tenant received_broadcast_version(ret=0, tenant_id=1003, version=1713878493426008) [2024-04-24 10:00:52.612486] INFO [SHARE.SCHEMA] refresh_tenant_schema (ob_multi_version_schema_service.cpp:2889) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=105] [REFRESH_SCHEMA] end refresh and add schema by tenant(ret=0, ret="OB_SUCCESS", tenant_id=1003, cost=1929) [2024-04-24 10:00:52.612570] INFO [SHARE.SCHEMA] refresh_tenant_schema (ob_multi_version_schema_service.cpp:2795) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=73] [REFRESH_SCHEMA] start to refresh and add schema by tenant(tenant_id=1004) [2024-04-24 10:00:52.614298] INFO [SHARE.SCHEMA] set_tenant_received_broadcast_version (ob_multi_version_schema_service.cpp:4100) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=66] try to set tenant received_broadcast_version(ret=0, tenant_id=1004, version=1713924050928872) [2024-04-24 10:00:52.614396] INFO [SHARE.SCHEMA] refresh_tenant_schema (ob_multi_version_schema_service.cpp:2889) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=86] [REFRESH_SCHEMA] end refresh and add schema by tenant(ret=0, ret="OB_SUCCESS", tenant_id=1004, cost=1747) [2024-04-24 10:00:52.614487] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2572) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=57] [REFRESH_SCHEMA] end refresh and add schema(ret=0, ret="OB_SUCCESS", tenant_ids=[], cost=12759) [2024-04-24 10:00:52.614593] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:308) [861415][SerScheQueue0][T0][YB420A000003-000616C2AAC92488-0-0] [lt=92] [REFRESH_SCHEMA] end refresh schema with new mode, (ret=0, ret="OB_SUCCESS", tenant_ids=[], used time=13175) [2024-04-24 10:00:52.621980] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [861686][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=147] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-24 10:00:52.626063] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [874532][T1004_LSArchive][T1004][YB420A000003-000616C2B568F0E7-0-0] [lt=78] gc stale ls task succ [2024-04-24 10:00:52.626115] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [862135][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=152] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}]) [2024-04-24 10:00:52.637019] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=97][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.637810] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=203] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.646404] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=108] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-04-24 10:00:52.646436] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [861685][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=103] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=11, size_used=16025, mem_used=27036672) [2024-04-24 10:00:52.653769] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=140] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.657841] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [862435][T1001_ObTimer][T1001][Y0-0000000000000000-0-0] [lt=86] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-04-24 10:00:52.658004] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [862435][T1001_ObTimer][T1001][Y0-0000000000000000-0-0] [lt=119] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-04-24 10:00:52.658067] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [862435][T1001_ObTimer][T1001][Y0-0000000000000000-0-0] [lt=57] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=9727) [2024-04-24 10:00:52.662718] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=197] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.671352] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.675906] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=78] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1003, min_version={val:1713924052029945001, v:0}, base_version={val:1713924051748148000, v:0}, server_count=2, svr_array=[{addr:"10.0.0.2:2882", version:{val:1713924052029945001, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1713924052625743, is_skipped:false}, {addr:"10.0.0.3:2882", version:{val:1713924052029945001, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1713924052675785, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2024-04-24 10:00:52.677809] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [862260][T1002_TenantWea][T1001][YB420A000003-000616C2B18CC97E-0-0] [lt=144] current clock is (clock_=0, max_seq_=0, mem_limit=192099116, cur_mem_hold=130023424, attr_.tenant_id_=1001) [2024-04-24 10:00:52.678200] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:157) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=266] submit log success(service_type_=0, last_id=1713924069270392000, limited_id=1713924069270392000) [2024-04-24 10:00:52.678200] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:157) [862260][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=132] submit log success(service_type_=0, last_id=1713924069338456000, limited_id=1713924069338456000) [2024-04-24 10:00:52.678284] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=77] submit log with lock success(service_type_=0, last_id=1713924069270392000, limited_id=1713924069270392000) [2024-04-24 10:00:52.678309] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [862260][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=103] submit log with lock success(service_type_=0, last_id=1713924069338456000, limited_id=1713924069338456000) [2024-04-24 10:00:52.681084] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [874526][T1004_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=72] is logging [2024-04-24 10:00:52.681201] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [874526][T1004_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=103] submit log with lock(ret=-4023, service_type_=0, last_id=1713924069270392000, limited_id=1713924069270392000) [2024-04-24 10:00:52.681764] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [861673][T1_LSMetaCh][T1][YB420A000003-000616C2A909ADA6-0-0] [lt=201] fetch root ls success(local_cluster_id=1713874979, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.2:2882", sql_port:2881, role:1, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:5, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713921656566585, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"10.0.0.3:2882", sql_port:2881, role:2, member_list:[{server:"10.0.0.2:2882", timestamp:1}, {server:"10.0.0.3:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:2, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["10.0.0.2:2882", "10.0.0.3:2882"]) [2024-04-24 10:00:52.683773] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=91] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.683914] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=128] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.683988] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=65] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.710785] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=75] ====== checkpoint timer task ====== [2024-04-24 10:00:52.710983] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=72] get_max_applied_scn(scn={val:1713924052129649001, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:5, palf_committed_end_lsn_:{lsn:314139808}, last_check_scn_:{val:1713924052677237002, v:0}, max_applied_cb_scn_:{val:1713924052129649001, v:0}}) [2024-04-24 10:00:52.711090] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1018) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=89] get_min_unreplayed_log_info in skip state(lsn={lsn:299748699}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713921656836981, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:299748699}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:299748699}, end_lsn:{lsn:299748699}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x14b14c9b9470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-04-24 10:00:52.711391] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=118] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.711495] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=92] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.711568] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=66] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.712213] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=133] succ to get rec scn(*this={this:0x14b14b804030, ls_id:{id:1}, tenant_id:1001, state:"L_WORKING", total_tx_ctx_count:2, active_tx_count:2, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1713895201571489000, v:0}, last_push_gc_task_ts_:1713895203315693, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:844}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-24 10:00:52.712328] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=102] tx ctx memtable get rec scn(this={ObITable:{this:0x14b14be62080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1713895203319956, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x14b14be62080, snapshot_version:{val:1713895203319956, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:1713924052605532000, v:0}) [2024-04-24 10:00:52.712418] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=77] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-04-24 10:00:52.712787] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=65] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1713895313405863001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1713895313405863001, v:0}}, }, *this={this:0x14b14c97a510, is_inited:true, freezing_scn:{val:1713877132047340000, v:0}, ls:{ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895203696037000, v:0}, clog_base_lsn:{lsn:134209536}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1713875407740689002, v:0}, all_id_meta:{id_meta:[{limited_id:1713924059338456000, latest_log_ts:{val:1713924042694950001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713898575276749001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1713910854505533001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:5, palf_env_:0x14b157ff2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-04-24 10:00:52.712982] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=180] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x14b14c988b90}, min_rec_scn={val:1713895203696037000, v:0}, ls_id_={id:1}) [2024-04-24 10:00:52.713085] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=84] get rec log scn(service_type_=0, rec_log_ts={val:1713924042694950001, v:0}) [2024-04-24 10:00:52.713185] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=94] get rec log scn(service_type_=1, rec_log_ts={val:1713898575276749001, v:0}) [2024-04-24 10:00:52.713250] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=59] get rec log scn(service_type_=2, rec_log_ts={val:1713910854505533001, v:0}) [2024-04-24 10:00:52.713328] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=64] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713895203696037000, v:0}, checkpoint_scn_in_ls_meta={val:1713895203696037000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-04-24 10:00:52.713402] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=62][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, curr_base_lsn={lsn:134209536}, new_base_lsn={lsn:134209536}, lsn={lsn:134209536}) [2024-04-24 10:00:52.713483] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=75] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:134209536}, ls->get_ls_id()={id:1}) [2024-04-24 10:00:52.713538] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [862432][T1001_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=50] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-04-24 10:00:52.715370] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [874529][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=91] ====== check clog disk timer task ====== [2024-04-24 10:00:52.715474] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [874529][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=82] get_disk_usage(ret=0, capacity(MB):=33177, used(MB):=390) [2024-04-24 10:00:52.715697] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [874529][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=135] cannot_recycle_log_size statistics(cannot_recycle_log_size=208136956, threshold=10436770529, need_update_checkpoint_scn=false) [2024-04-24 10:00:52.718029] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [862263][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=156] ====== check clog disk timer task ====== [2024-04-24 10:00:52.718108] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [862263][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=69] get_disk_usage(ret=0, capacity(MB):=5529, used(MB):=1369) [2024-04-24 10:00:52.718220] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [862263][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=66] cannot_recycle_log_size statistics(cannot_recycle_log_size=898948009, threshold=1739461755, need_update_checkpoint_scn=false) [2024-04-24 10:00:52.718546] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [862276][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=147] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}, {id:1002}]) [2024-04-24 10:00:52.728679] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1457) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2B1BC3310-0-0] [lt=85] receive register_req from acceptor, ignore(self="10.0.0.3:2882", palf_id=1, child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924052724061, keepalive_ts:-1}) [2024-04-24 10:00:52.728789] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1476) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2B1BC3310-0-0] [lt=96] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="10.0.0.3:2882", child={server:"10.0.0.2:2882", region:"sys_region", register_time_us:1713924052724061, keepalive_ts:-1}, is_to_leader=true) [2024-04-24 10:00:52.732582] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD2AE-0-0] [lt=136][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=330741, log_task={header:{begin_lsn:{lsn:88132621}, end_lsn:{lsn:88132743}, log_id:330741, min_scn:{val:1713924052130828000, v:0}, max_scn:{val:1713924052130828000, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:88132621}, data_checksum:2222604892, accum_checksum:2054469505, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052182701, freeze_ts:1713924052182718, submit_ts:1713924052182941, flushed_ts:1713924052568351, gen_to_freeze cost time:17, gen_to_submit cost time:240, submit_to_flush cost time:385410}, fs_cb_begin_ts=1713924052732531, log_life_time=549830) [2024-04-24 10:00:52.733401] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=149] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8907464704, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:52.733503] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0536-0-0] [lt=140][errcode=-4023] tx is 2pc logging(this={this:0x14b148003290, ref:4, trans_id:{txid:176931}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924082541480, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052572154], ctx_create_time:1713924052549745}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79642727}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052556207, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052568961000, v:0}, log_ts:{val:1713924052568961000, v:0}, lsn:{lsn:79643010}, submit_ts:1713924052568961}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176931}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176931}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924052568961000, v:0}, start_scn:{val:1713924052225180002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465782000 trans_version={val:1713924052568961000, v:0} commit_version={val:1713924052568961000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176931} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3990623770 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052225180002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052568961, block_frozen_memtable:null}) [2024-04-24 10:00:52.733734] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0536-0-0] [lt=219][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b148003290, ref:4, trans_id:{txid:176931}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924082541480, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052572154], ctx_create_time:1713924052549745}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[{lsn:79642727}], redo_log_no:1, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:2, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052556207, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():2, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052568961000, v:0}, log_ts:{val:1713924052568961000, v:0}, lsn:{lsn:79643010}, submit_ts:1713924052568961}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176931}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176931}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1713924052568961000, v:0}, start_scn:{val:1713924052225180002, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465782000 trans_version={val:1713924052568961000, v:0} commit_version={val:1713924052568961000, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176931} ls_id=1 callback_alloc_count=2 callback_free_count=0 checksum=3990623770 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=3 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=3 unsynced_cnt=2 unsubmitted_cnt_=0 cb_statistics:[main=3, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052225180002, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052568961, block_frozen_memtable:null}) [2024-04-24 10:00:52.733862] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0536-0-0] [lt=122][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176931}) [2024-04-24 10:00:52.733987] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0536-0-0] [lt=115][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176931}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924052729037, cluster_id:1713874979}, expire_ts:1713924082565418, commit_start_scn:{val:1713924052558792000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:52.734092] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BA1C0536-0-0] [lt=81][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176931}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:6, timestamp:1713924052729037, cluster_id:1713874979}, expire_ts:1713924082565418, commit_start_scn:{val:1713924052558792000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:52.734244] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=140][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:79642671}], scn_array=[{val:1713924052225180002, v:0}], curr_size=2549, accum_size=17671, time_cost=160361) [2024-04-24 10:00:52.734678] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=72][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:40362599}], scn_array=[{val:1713924052251340000, v:0}], curr_size=449, accum_size=1142, time_cost=161340) [2024-04-24 10:00:52.734766] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862144][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=125][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:165107787}], scn_array=[{val:1713924052510325000, v:0}], curr_size=1573, accum_size=3390, time_cost=161427) [2024-04-24 10:00:52.734956] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=125][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67161839}], scn_array=[{val:1713924052426649000, v:0}], curr_size=122, accum_size=1630, time_cost=160634) [2024-04-24 10:00:52.736405] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [874703][T1003_ReqMemEvi][T1003][Y0-0000000000000000-0-0] [lt=67] run rewrite rule refresh task(rule_mgr_->tenant_id_=1003) [2024-04-24 10:00:52.737104] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=105][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:298531058}], scn_array=[{val:1713924052426649000, v:0}], curr_size=122, accum_size=3730, time_cost=162775) [2024-04-24 10:00:52.737269] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=151][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179756117}], scn_array=[{val:1713924052520292000, v:0}], curr_size=1034, accum_size=6405, time_cost=162227) [2024-04-24 10:00:52.737331] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=97][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.737345] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862143][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=140][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:1002202223}], scn_array=[{val:1713924052426649000, v:0}], curr_size=122, accum_size=5084, time_cost=125727) [2024-04-24 10:00:52.737961] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=177][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:314139808}], scn_array=[{val:1713924052605532000, v:0}], curr_size=542, accum_size=4414, time_cost=127598) [2024-04-24 10:00:52.738024] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=165] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.738840] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [874703][T1003_ReqMemEvi][T1003][YB420A000003-000616C2B7692782-0-0] [lt=82] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924041284274, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924052738743, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:52.739178] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [874703][T1003_ReqMemEvi][T1003][Y0-0000000000000000-0-0] [lt=147] is logging [2024-04-24 10:00:52.739260] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [874703][T1003_ReqMemEvi][T1003][Y0-0000000000000000-0-0] [lt=71] submit log with lock(ret=-4023, service_type_=0, last_id=1713924069270392000, limited_id=1713924069270392000) [2024-04-24 10:00:52.739353] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [874703][T1003_ReqMemEvi][T1003][Y0-0000000000000000-0-0] [lt=84] get number(ret=0, service_type_=0, range=1, base_id=1713924052738743000, start_id=1713924052738743000, end_id=1713924052738743001) [2024-04-24 10:00:52.739362] INFO do_work (ob_rl_mgr.cpp:709) [861413][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=76] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-24 10:00:52.754003] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=153] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.761729] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [861648][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=92] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-04-24 10:00:52.762936] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=93] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.771538] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=63][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.772122] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD2AF-0-0] [lt=117][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=299924, log_task={header:{begin_lsn:{lsn:40362599}, end_lsn:{lsn:40363048}, log_id:299924, min_scn:{val:1713924052251340000, v:0}, max_scn:{val:1713924052251340000, v:0}, data_len:393, proposal_id:5, prev_lsn:{lsn:40362477}, prev_proposal_id:5, committed_end_lsn:{lsn:40362599}, data_checksum:225108780, accum_checksum:94826471, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924052571908, freeze_ts:1713924052571917, submit_ts:1713924052571958, flushed_ts:1713924052734946, gen_to_freeze cost time:9, gen_to_submit cost time:50, submit_to_flush cost time:162988}, fs_cb_begin_ts=1713924052772068, log_life_time=200160) [2024-04-24 10:00:52.772208] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1208) [874559][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=143] succ to submit log task to replay service(to_submit_lsn={lsn:40362655}, to_submit_scn={val:1713924052251340000, v:0}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713924052772075, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:2137654, ret_code_:0}}, next_to_submit_lsn_:{lsn:40363048}, next_to_submit_scn_:{val:1713924052251340001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1713877500304026000, v:0}, iterator_:{iterator_impl:{buf_:0x14b07d217000, next_round_pread_size:3678208, curr_read_pos:449, curr_read_buf_start_pos:0, curr_read_buf_end_pos:449, log_storage_:{IteratorStorage:{start_lsn:{lsn:40362599}, end_lsn:{lsn:40363048}, read_buf:{buf_len_:53248, buf_:0x14b07d217000}, block_size:67104768, log_storage_:0x14b0c0efd470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713924052251340000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:361, scn_:{val:1713924052251340000, v:0}, data_checksum:2401667391, flag:0}}, init_mode_version:0, accumulate_checksum:94826471, curr_entry_is_padding:0, padding_entry_size:393, padding_entry_scn:{val:1713924052251340000, v:0}}}}}) [2024-04-24 10:00:52.772798] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD2B0-0-0] [lt=86][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=304872, log_task={header:{begin_lsn:{lsn:79641736}, end_lsn:{lsn:79642671}, log_id:304872, min_scn:{val:1713924052225180000, v:0}, max_scn:{val:1713924052225180001, v:0}, data_len:879, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:79641736}, data_checksum:2917663955, accum_checksum:4118267575, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052225646, freeze_ts:1713924052504929, submit_ts:1713924052505102, flushed_ts:1713924052573270, gen_to_freeze cost time:279283, gen_to_submit cost time:279456, submit_to_flush cost time:68168}, fs_cb_begin_ts=1713924052772750, log_life_time=547104) [2024-04-24 10:00:52.772946] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C5D-0-0] [lt=89][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=259327, log_task={header:{begin_lsn:{lsn:67161839}, end_lsn:{lsn:67161961}, log_id:259327, min_scn:{val:1713924052426649000, v:0}, max_scn:{val:1713924052426649000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:67161839}, data_checksum:667309872, accum_checksum:177175257, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052510982, freeze_ts:1713924052573922, submit_ts:1713924052574084, flushed_ts:1713924052735258, gen_to_freeze cost time:62940, gen_to_submit cost time:63102, submit_to_flush cost time:161174}, fs_cb_begin_ts=1713924052772860, log_life_time=261878) [2024-04-24 10:00:52.772979] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874553][T1004_ApplySrv1][T1004][YB420A000003-000616C2BFC8F2C2-0-0] [lt=109][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052225180000, v:0}, log_ts:{val:1713924052225180000, v:0}, lsn:{lsn:79641792}, submit_ts:1713924052225180}, this:0x14b14807de40, is_inited_:true, trans_id:{txid:176929}, ls_id:{id:1}, ctx:0x14b14807aed0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=547782, ctx_lock_wait_time=7) [2024-04-24 10:00:52.773711] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C5E-0-0] [lt=127][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=301044, log_task={header:{begin_lsn:{lsn:165107787}, end_lsn:{lsn:165109360}, log_id:301044, min_scn:{val:1713924052510325000, v:0}, max_scn:{val:1713924052510325000, v:0}, data_len:1517, proposal_id:5, prev_lsn:{lsn:165107665}, prev_proposal_id:5, committed_end_lsn:{lsn:165107787}, data_checksum:895030897, accum_checksum:525959152, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924052572388, freeze_ts:1713924052572396, submit_ts:1713924052572442, flushed_ts:1713924052735056, gen_to_freeze cost time:8, gen_to_submit cost time:54, submit_to_flush cost time:162614}, fs_cb_begin_ts=1713924052773655, log_life_time=201267) [2024-04-24 10:00:52.773972] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325892][T1_L0_G1][T1][YB420A000002-000616C2AADD1423-0-0] [lt=76][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=364252, log_task={header:{begin_lsn:{lsn:179755995}, end_lsn:{lsn:179756117}, log_id:364252, min_scn:{val:1713924052284612000, v:0}, max_scn:{val:1713924052284612000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:179755457}, prev_proposal_id:5, committed_end_lsn:{lsn:179755457}, data_checksum:667309872, accum_checksum:1293637833, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924052325615, freeze_ts:1713924052325622, submit_ts:1713924052325726, flushed_ts:1713924052571629, gen_to_freeze cost time:7, gen_to_submit cost time:111, submit_to_flush cost time:245903}, fs_cb_begin_ts=1713924052773848, log_life_time=448233) [2024-04-24 10:00:52.774835] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [862285][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=112] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924042608941, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924052773937, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:52.775078] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB90A-0-0] [lt=125][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=505473, log_task={header:{begin_lsn:{lsn:314139808}, end_lsn:{lsn:314140350}, log_id:505473, min_scn:{val:1713924052605532000, v:0}, max_scn:{val:1713924052605532000, v:0}, data_len:486, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:314139808}, data_checksum:2076119069, accum_checksum:791551159, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052606470, freeze_ts:1713924052606478, submit_ts:1713924052606629, flushed_ts:1713924052739336, gen_to_freeze cost time:8, gen_to_submit cost time:159, submit_to_flush cost time:132707}, fs_cb_begin_ts=1713924052775015, log_life_time=168545) [2024-04-24 10:00:52.775207] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=139][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052605532000, v:0}, log_ts:{val:1713924052605532000, v:0}, lsn:{lsn:314139864}, submit_ts:1713924052605532}, this:0x14b15a178440, is_inited_:true, trans_id:{txid:568583}, ls_id:{id:1}, ctx:0x14b15a1754d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=169652, ctx_lock_wait_time=17) [2024-04-24 10:00:52.775291] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F84-0-0] [lt=172][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=482566, log_task={header:{begin_lsn:{lsn:298531058}, end_lsn:{lsn:298531180}, log_id:482566, min_scn:{val:1713924052426649000, v:0}, max_scn:{val:1713924052426649000, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:298531058}, data_checksum:667309872, accum_checksum:1107073625, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052562548, freeze_ts:1713924052573936, submit_ts:1713924052574118, flushed_ts:1713924052737492, gen_to_freeze cost time:11388, gen_to_submit cost time:11570, submit_to_flush cost time:163374}, fs_cb_begin_ts=1713924052775245, log_life_time=212697) [2024-04-24 10:00:52.776034] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C5F-0-0] [lt=111][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=673432, log_task={header:{begin_lsn:{lsn:1002202223}, end_lsn:{lsn:1002202345}, log_id:673432, min_scn:{val:1713924052426649000, v:0}, max_scn:{val:1713924052426649000, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1002202223}, data_checksum:2222604892, accum_checksum:577696149, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052611337, freeze_ts:1713924052611345, submit_ts:1713924052611451, flushed_ts:1713924052737626, gen_to_freeze cost time:8, gen_to_submit cost time:114, submit_to_flush cost time:126175}, fs_cb_begin_ts=1713924052775983, log_life_time=164646) [2024-04-24 10:00:52.777384] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=92] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1713924051825760000, v:0}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=951606, in_cluster_service=true, cluster_version={val:1713924052035048000, v:0}, min_cluster_version={val:1713924052035048000, v:0}, max_cluster_version={val:1713924052035048000, v:0}, get_cluster_version_err=0, cluster_version_delta=742318, cluster_service_master="10.0.0.3:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=348150, succ_cluster_heartbeat_count=66, cluster_heartbeat_interval=50000, local_cluster_version={val:1713923391817005001, v:0}, local_cluster_delta=660960361, force_self_check=false, weak_read_refresh_interval=100000) [2024-04-24 10:00:52.780342] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=92] current clock is (clock_=0, max_seq_=0, mem_limit=3181127926, cur_mem_hold=297795584, attr_.tenant_id_=1004) [2024-04-24 10:00:52.780436] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=85] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924042314454, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924052777450, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:52.780606] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=191] new mysql sessid created(conn.sessid_=3221542566, support_ssl=false) [2024-04-24 10:00:52.780775] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=122] sm conn init succ(conn.sessid_=3221542566, sess.client_addr_="10.0.0.3:48658") [2024-04-24 10:00:52.780775] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:860) [2243643][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=75] memtable handle rollback to successfuly(from_seq_no=1713924052777454, to_seq_no=1713924052776271, *this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713890377730352 max_table_version=1713890377730352 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:877478} ls_id=1001 callback_alloc_count=0 callback_free_count=0 checksum=0 tmp_checksum=0 checksum_scn={val:0, v:0} redo_filled_count=0 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=0 unsynced_cnt=0 unsubmitted_cnt_=0 cb_statistics:[main=1, slave=0, merge=0, tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}) [2024-04-24 10:00:52.780950] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=82] accept one succ(*s={this:0x14b0c2204030, session_id:3221542566, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:52.781514] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:943) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=110][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x14b0c2204030, session_id:3221542566, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:52.781716] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=118] kill and revert session(conn.sessid_=3221542566, proxy_sessid=0, server_id=1, ret=0) [2024-04-24 10:00:52.781839] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:918) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=116] sock ref clean, do destroy(*s={this:0x14b0c2204030, session_id:3221542566, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-24 10:00:52.782050] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:223) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=197] mark session id unused(conn.sessid_=3221542566) [2024-04-24 10:00:52.782127] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [861518][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=70] connection close(sessid=3221542566, proxy_sessid=0, tenant_id=0, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:0], disconnect_state=0) [2024-04-24 10:00:52.783074] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [875191][T1004_L0_G5][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=115] get number(ret=0, service_type_=0, range=1, base_id=1713924052777450000, start_id=1713924052777450000, end_id=1713924052777450001) [2024-04-24 10:00:52.784561] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=81] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.784657] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [861569][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=62][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:179757151}], scn_array=[{val:1713924052609419000, v:0}], curr_size=1045, accum_size=7450, time_cost=10280) [2024-04-24 10:00:52.785031] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=96] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.785182] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=140] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.785217] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [861679][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=163] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=1, total_destructed_detector=1, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x14b1987ea2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-04-24 10:00:52.787010] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [2243678][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=80] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924042332037, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924052785448, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:52.788765] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:156) [2247495][T1002_L0_G0][T1002][YB420A000002-000616C2BF5CC88C-0-0] [lt=230] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_time=1713924042420012, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1713924052788623, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0}) [2024-04-24 10:00:52.789062] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=82] get number(ret=0, service_type_=0, range=1, base_id=1713924052788623000, start_id=1713924052788623000, end_id=1713924052788623001) [2024-04-24 10:00:52.791831] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [862476][T1001_L0_G5][T1001][YB420A000002-000616C2BEDDD138-0-0] [lt=140] is logging [2024-04-24 10:00:52.792072] WDIAG [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:116) [862476][T1001_L0_G5][T1001][YB420A000002-000616C2BEDDD138-0-0] [lt=218][errcode=-4023] submit log callback use too mush time(submit_log_ts=1713924052678194, cb={log_ts:{val:1713924052677237001, v:0}, log_buf:"", pos:40, id_srv_type:0, limited_id:1713924069338456000}, service_type=0, latest_log_ts={val:1713924042694950001, v:0}, self="10.0.0.3:2882") [2024-04-24 10:00:52.792236] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [862476][T1001_L0_G5][T1001][YB420A000002-000616C2BEDDD138-0-0] [lt=156] submit log with lock(ret=-4023, service_type_=0, last_id=1713924069338456000, limited_id=1713924069338456000) [2024-04-24 10:00:52.792340] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [874796][T1003_L0_G5][T1003][YB420A000002-000616C2BEDDD13A-0-0] [lt=95] is logging [2024-04-24 10:00:52.792486] WDIAG [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:116) [874796][T1003_L0_G5][T1003][YB420A000002-000616C2BEDDD13A-0-0] [lt=104][errcode=-4023] submit log callback use too mush time(submit_log_ts=1713924052678194, cb={log_ts:{val:1713924052677237001, v:0}, log_buf:"", pos:40, id_srv_type:0, limited_id:1713924069270392000}, service_type=0, latest_log_ts={val:1713924042654251001, v:0}, self="10.0.0.3:2882") [2024-04-24 10:00:52.792571] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [874796][T1003_L0_G5][T1003][YB420A000002-000616C2BEDDD13A-0-0] [lt=77] submit log with lock(ret=-4023, service_type_=0, last_id=1713924069270392000, limited_id=1713924069270392000) [2024-04-24 10:00:52.811799] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=138] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.811940] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=128] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.812027] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=80] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.837500] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=78][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.838492] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=77] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.839345] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [861692][T1_LSArchiveMgr][T1][YB420A000003-000616C2A888F0E7-0-0] [lt=86] gc stale ls task succ [2024-04-24 10:00:52.840376] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:788) [862091][T1_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=133] ps cache evict(tenant_id=1, stmt_id_map_.size()=15, stmt_info_map_.size()=15, op.get_used_size()=32344, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=15) [2024-04-24 10:00:52.846750] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=111] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-04-24 10:00:52.854176] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=83] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.861016] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [862417][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=104] report RowHolderMapper summary info(count=1, bkt_cnt=248) [2024-04-24 10:00:52.863121] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=116] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.871805] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=73][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.876970] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:426) [874701][T1003_FreInfoRe][T1003][Y0-0000000000000000-0-0] [lt=88] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2024-04-24 10:00:52.879316] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [862431][T1001_TenantWea][T1001][YB420A000003-000616C2AEECE7F3-0-0] [lt=106] current clock is (clock_=0, max_seq_=0, mem_limit=192099116, cur_mem_hold=130023424, attr_.tenant_id_=1001) [2024-04-24 10:00:52.879541] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=84] is logging [2024-04-24 10:00:52.879627] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [862431][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=63] submit log with lock(ret=-4023, service_type_=0, last_id=1713924069338456000, limited_id=1713924069338456000) [2024-04-24 10:00:52.881630] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [862131][T1002_Occam][T1002][YB420A000003-000616C2AFD988F6-0-0] [lt=134] for each ls(succ_num=3, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:52.883063] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [874745][T1003_ArcSender][T1003][YB420A000003-000616C2B638F0E7-0-0] [lt=90][errcode=-4018] try free send task failed(ret=-4018) [2024-04-24 10:00:52.884648] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=107] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.884766] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=100] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.884853] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=70] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.886128] INFO [STORAGE] schedule_all_tablets_minor (ob_tenant_tablet_scheduler.cpp:511) [861698][T1_MergeLoop][T1][YB420A000003-000616C2ABA8F27A-0-0] [lt=99] start schedule all tablet minor merge(minor_ls_tablet_iter_={ls_idx:-1, ls_ids:[{id:1}], tablet_idx:0, tablet_ids_.count():0, schedule_tablet_cnt:0, max_batch_tablet_cnt:50000, report_scn_flag:false}) [2024-04-24 10:00:52.898008] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [861461][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=303] skip inspect bad block(last_check_time=1713875167993818, last_macro_idx=-1) [2024-04-24 10:00:52.909268] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862142][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=80][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:67161961}], scn_array=[{val:1713924052792518000, v:0}], curr_size=122, accum_size=1752, time_cost=97288) [2024-04-24 10:00:52.910607] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD2B1-0-0] [lt=148][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=330742, log_task={header:{begin_lsn:{lsn:88132743}, end_lsn:{lsn:88135122}, log_id:330742, min_scn:{val:1713924052448472000, v:0}, max_scn:{val:1713924052448472000, v:0}, data_len:2323, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:88132621}, data_checksum:1925707006, accum_checksum:1099793869, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052449579, freeze_ts:1713924052568389, submit_ts:1713924052568507, flushed_ts:1713924052575504, gen_to_freeze cost time:118810, gen_to_submit cost time:118928, submit_to_flush cost time:6997}, fs_cb_begin_ts=1713924052910533, log_life_time=460954) [2024-04-24 10:00:52.910802] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2C6-0-0] [lt=285][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052448472000, v:0}, log_ts:{val:1713924052448472000, v:0}, lsn:{lsn:88132799}, submit_ts:1713924052448472}, this:0x14b1481b8b40, is_inited_:true, trans_id:{txid:327215}, ls_id:{id:1001}, ctx:0x14b1481b5bd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=462306, ctx_lock_wait_time=14) [2024-04-24 10:00:52.911144] INFO [LIB] stat (utility.h:1197) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2D0-0-0] [lt=169] [APPLY STAT CB APPEND COST TIME](cur_stat_count=5, stat_interval=5000000, avg (count/cost)=138, this=0x14b0c0e3c280, extra_info="") [2024-04-24 10:00:52.911250] INFO [LIB] stat (utility.h:1197) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2D0-0-0] [lt=82] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=5, stat_interval=5000000, avg (count/cost)=393100, this=0x14b0c0e3c4b0, extra_info="") [2024-04-24 10:00:52.911368] INFO [LIB] stat (utility.h:1197) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2D0-0-0] [lt=101] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=5, stat_interval=5000000, avg (count/cost)=0, this=0x14b0c0e3c6e0, extra_info="") [2024-04-24 10:00:52.911469] INFO [LIB] stat (utility.h:1197) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2D0-0-0] [lt=80] [APPLY STAT CB EXECUTE TIME](cur_stat_count=5, stat_interval=5000000, avg (count/cost)=4, this=0x14b0c0e3c910, extra_info="") [2024-04-24 10:00:52.911555] INFO [LIB] stat (utility.h:1197) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2D0-0-0] [lt=74] [APPLY STAT CB TOTAL TIME](cur_stat_count=5, stat_interval=5000000, avg (count/cost)=393242, this=0x14b0c0e3cb40, extra_info="") [2024-04-24 10:00:52.911795] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=147] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.911996] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [862320][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=75][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:314140964}], scn_array=[{val:1713924052677237003, v:0}], curr_size=122, accum_size=5150, time_cost=73420) [2024-04-24 10:00:52.912204] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=122] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.912308] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=86] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.919228] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326860][T1001_L0_G1][T1001][YB420A000002-000616C2BD9FB90B-0-0] [lt=85][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=505474, log_task={header:{begin_lsn:{lsn:314140350}, end_lsn:{lsn:314140964}, log_id:505474, min_scn:{val:1713924052677237001, v:0}, max_scn:{val:1713924052677237002, v:0}, data_len:558, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:314139808}, data_checksum:3161483856, accum_checksum:432881646, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052678162, freeze_ts:1713924052739353, submit_ts:1713924052739462, flushed_ts:1713924052749326, gen_to_freeze cost time:61191, gen_to_submit cost time:61300, submit_to_flush cost time:9864}, fs_cb_begin_ts=1713924052919156, log_life_time=240994) [2024-04-24 10:00:52.919385] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:681) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=90] update id meta(ret=0, service_type=0, limited_id=1713924069338456000, latest_scn={val:1713924052677237001, v:0}, *this={tenant_id:1001, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895203696037000, v:0}, clog_base_lsn:{lsn:134209536}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1713875407740689002, v:0}, all_id_meta:{id_meta:[{limited_id:1713924069338456000, latest_log_ts:{val:1713924052677237001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713898575276749001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1713910854505533001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}) [2024-04-24 10:00:52.919544] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326783][T1003_L0_G1][T1003][YB420A000002-000616C2BE9F5F85-0-0] [lt=120][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=482567, log_task={header:{begin_lsn:{lsn:298531180}, end_lsn:{lsn:298532280}, log_id:482567, min_scn:{val:1713924052677237001, v:0}, max_scn:{val:1713924052681021000, v:0}, data_len:1044, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:298531058}, data_checksum:4092182050, accum_checksum:2238828560, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052678162, freeze_ts:1713924052737504, submit_ts:1713924052737633, flushed_ts:1713924052747734, gen_to_freeze cost time:59342, gen_to_submit cost time:59471, submit_to_flush cost time:10101}, fs_cb_begin_ts=1713924052919480, log_life_time=241318) [2024-04-24 10:00:52.919556] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:189) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=146] handle submit callback(service_type_=0, success=true, used_ts=241350, log_ts={val:1713924052677237001, v:0}, limited_id=1713924069338456000) [2024-04-24 10:00:52.919705] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:561) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=137] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2024-04-24 10:00:52.919680] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=77][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052681021000, v:0}, log_ts:{val:1713924052681021000, v:0}, lsn:{lsn:298531794}, submit_ts:1713924052681021}, this:0x14b0fbee5680, is_inited_:true, trans_id:{txid:548745}, ls_id:{id:1}, ctx:0x14b0fbee2710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=238603, ctx_lock_wait_time=6) [2024-04-24 10:00:52.919790] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:599) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=76] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2024-04-24 10:00:52.919930] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:681) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=101] update id meta(ret=0, service_type=0, limited_id=1713924069270392000, latest_scn={val:1713924052677237001, v:0}, *this={tenant_id:1003, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1713895201246660000, v:0}, clog_base_lsn:{lsn:67104768}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1713877340363486001, v:0}, all_id_meta:{id_meta:[{limited_id:1713924069270392000, latest_log_ts:{val:1713924052677237001, v:0}}, {limited_id:2000001, latest_log_ts:{val:1713900544098309001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1713877439572285001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}) [2024-04-24 10:00:52.920006] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862408][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=140][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052677237000, v:0}, log_ts:{val:1713924052677237002, v:0}, lsn:{lsn:314140478}, submit_ts:1713924052677237}, this:0x14b15a16e000, is_inited_:true, trans_id:{txid:568584}, ls_id:{id:1}, ctx:0x14b15a16b090, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=242748, ctx_lock_wait_time=13) [2024-04-24 10:00:52.920047] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:189) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=95] handle submit callback(service_type_=0, success=true, used_ts=241845, log_ts={val:1713924052677237001, v:0}, limited_id=1713924069270392000) [2024-04-24 10:00:52.920136] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:561) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=81] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2024-04-24 10:00:52.920228] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:599) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=81] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2024-04-24 10:00:52.920420] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=91][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052677237000, v:0}, log_ts:{val:1713924052677237002, v:0}, lsn:{lsn:298531308}, submit_ts:1713924052677237}, this:0x14b0fc124480, is_inited_:true, trans_id:{txid:548744}, ls_id:{id:1}, ctx:0x14b0fc121510, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=243160, ctx_lock_wait_time=14) [2024-04-24 10:00:52.920624] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874712][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=94][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052738743000, v:0}, log_ts:{val:1713924052738743000, v:0}, lsn:{lsn:298532336}, submit_ts:1713924052738743}, this:0x14b0fbfc3d40, is_inited_:true, trans_id:{txid:548746}, ls_id:{id:1}, ctx:0x14b0fbfc0dd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=181856, ctx_lock_wait_time=15) [2024-04-24 10:00:52.920947] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF2DD2B2-0-0] [lt=151][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=304873, log_task={header:{begin_lsn:{lsn:79642671}, end_lsn:{lsn:79645220}, log_id:304873, min_scn:{val:1713924052225180002, v:0}, max_scn:{val:1713924052568961000, v:0}, data_len:2493, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:79641736}, data_checksum:2522736233, accum_checksum:1019636474, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052572218, freeze_ts:1713924052573304, submit_ts:1713924052573561, flushed_ts:1713924052734776, gen_to_freeze cost time:1086, gen_to_submit cost time:1343, submit_to_flush cost time:161215}, fs_cb_begin_ts=1713924052920850, log_life_time=348632) [2024-04-24 10:00:52.921345] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59CC-0-0] [lt=90][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=299925, log_task={header:{begin_lsn:{lsn:40363048}, end_lsn:{lsn:40363170}, log_id:299925, min_scn:{val:1713924052558792000, v:0}, max_scn:{val:1713924052558792000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:40362599}, prev_proposal_id:5, committed_end_lsn:{lsn:40363048}, data_checksum:1715074492, accum_checksum:1542984455, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924052771974, freeze_ts:1713924052771982, submit_ts:1713924052772034, flushed_ts:1713924052779117, gen_to_freeze cost time:8, gen_to_submit cost time:60, submit_to_flush cost time:7083}, fs_cb_begin_ts=1713924052921303, log_life_time=149329) [2024-04-24 10:00:52.921979] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874554][T1004_ApplySrv2][T1004][YB420A000003-000616C2BFC8F2CA-0-0] [lt=471][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:0, v:0}, log_ts:{val:1713924052225180002, v:0}, lsn:{lsn:79642727}, submit_ts:1713924052568961}, this:0x14b148005ae8, is_inited_:true, trans_id:{txid:176931}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=353001, ctx_lock_wait_time=6) [2024-04-24 10:00:52.922077] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874500][T1004_ApplySrv0][T1004][YB420A000003-000616C2BFC8F2A2-0-0] [lt=189][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052568961000, v:0}, log_ts:{val:1713924052568961000, v:0}, lsn:{lsn:79643010}, submit_ts:1713924052568961}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176931}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=353045, ctx_lock_wait_time=65) [2024-04-24 10:00:52.922373] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [862260][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=98] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1713924052426649000, v:0}, total_part_count:3, valid_inner_part_count:1, valid_user_part_count:1}, server_version_delta=495708, in_cluster_service=true, cluster_version={val:1713924052017398000, v:0}, min_cluster_version={val:1713924052017398000, v:0}, max_cluster_version={val:1713924052017398000, v:0}, get_cluster_version_err=0, cluster_version_delta=904959, cluster_service_master="10.0.0.3:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=335081, succ_cluster_heartbeat_count=12, cluster_heartbeat_interval=50000, local_cluster_version={val:1713923632940224000, v:0}, local_cluster_delta=419982133, force_self_check=false, weak_read_refresh_interval=100000) [2024-04-24 10:00:52.925858] INFO [SQL.JO] compute_table_location (ob_join_order.cpp:278) [861893][T1_L0_G0][T1][YB420A000003-000616C2ADD8FCD9-0-0] [lt=84] succeed to calculate base table sharding info(table_id=18446744073709551614, ref_table_id=104, is_global_index=false) [2024-04-24 10:00:52.926013] INFO [SQL.JO] init_est_sel_info_for_access_path (ob_join_order.cpp:12189) [861893][T1_L0_G0][T1][YB420A000003-000616C2ADD8FCD9-0-0] [lt=135] total rowcount, use statistics(table_meta_info_.table_row_count_=10, table_meta_info_.average_row_size_=2.060000000000000000e+02, table_meta_info_.micro_block_count_=1, table_meta_info_.part_size_=2.060000000000000000e+03) [2024-04-24 10:00:52.927464] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2325892][T1_L0_G1][T1][YB420A000002-000616C2A9FC0BDA-0-0] [lt=116][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=364254, log_task={header:{begin_lsn:{lsn:179757151}, end_lsn:{lsn:179758196}, log_id:364254, min_scn:{val:1713924052609419000, v:0}, max_scn:{val:1713924052761265000, v:0}, data_len:989, proposal_id:5, prev_lsn:{lsn:179756117}, prev_proposal_id:5, committed_end_lsn:{lsn:179757151}, data_checksum:2971201583, accum_checksum:3563207797, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924052773736, freeze_ts:1713924052773752, submit_ts:1713924052773813, flushed_ts:1713924052784950, gen_to_freeze cost time:16, gen_to_submit cost time:77, submit_to_flush cost time:11137}, fs_cb_begin_ts=1713924052927403, log_life_time=153667) [2024-04-24 10:00:52.929610] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=82] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-24 10:00:52.929814] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:390) [861494][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=158] get number(ret=0, service_type_=0, range=1, base_id=1713924052929711000, start_id=1713924052929711000, end_id=1713924052929711001) [2024-04-24 10:00:52.930033] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=161] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1713924052929776], gts:1713924052929711000, latest_srr:[mts=1713924052929776]}) [2024-04-24 10:00:52.930260] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [861494][TsMgr][T0][Y0-0000000000000000-0-0] [lt=123] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1713924052930165], gts:1713924052929711000, latest_srr:[mts=1713924052930165]}) [2024-04-24 10:00:52.933424] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874407][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=128][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:40363170}], scn_array=[{val:1713924052792518000, v:0}], curr_size=122, accum_size=1386, time_cost=11992) [2024-04-24 10:00:52.933998] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C60-0-0] [lt=98][errcode=-4389] log_task life cost too much time(palf_id=1001, self="10.0.0.3:2882", log_id=673433, log_task={header:{begin_lsn:{lsn:1002202345}, end_lsn:{lsn:1002203918}, log_id:673433, min_scn:{val:1713924052792518000, v:0}, max_scn:{val:1713924052792518000, v:0}, data_len:1517, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1002202345}, data_checksum:1357170248, accum_checksum:1223887000, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052792952, freeze_ts:1713924052792960, submit_ts:1713924052793121, flushed_ts:1713924052799346, gen_to_freeze cost time:8, gen_to_submit cost time:169, submit_to_flush cost time:6225}, fs_cb_begin_ts=1713924052933956, log_life_time=141004) [2024-04-24 10:00:52.934498] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2BF3E8C61-0-0] [lt=175][errcode=-4389] log_task life cost too much time(palf_id=1, self="10.0.0.3:2882", log_id=259328, log_task={header:{begin_lsn:{lsn:67161961}, end_lsn:{lsn:67162083}, log_id:259328, min_scn:{val:1713924052792518000, v:0}, max_scn:{val:1713924052792518000, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:67161961}, data_checksum:667309872, accum_checksum:289051122, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1713924052811590, freeze_ts:1713924052811610, submit_ts:1713924052811743, flushed_ts:1713924052909470, gen_to_freeze cost time:20, gen_to_submit cost time:153, submit_to_flush cost time:97727}, fs_cb_begin_ts=1713924052934430, log_life_time=122840) [2024-04-24 10:00:52.934769] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=96] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8892784640, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:52.935273] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=162][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052792518000, v:0}, log_ts:{val:1713924052792518000, v:0}, lsn:{lsn:1002202401}, submit_ts:1713924052792518}, this:0x14b168f30680, is_inited_:true, trans_id:{txid:877478}, ls_id:{id:1001}, ctx:0x14b168f2d710, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=142741, ctx_lock_wait_time=7) [2024-04-24 10:00:52.937719] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=75][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.938451] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=104] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.940976] WDIAG [STORAGE.TRANS] test_lock (ob_trans_ctx.cpp:245) [874555][T1004_ApplySrv3][T1004][YB420A000003-000616C2BFC8F2D0-0-0] [lt=58][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1713924052788623000, v:0}, log_ts:{val:1713924052788623000, v:0}, lsn:{lsn:79645276}, submit_ts:1713924052788623}, this:0x14b055e38fc0, is_inited_:true, trans_id:{txid:176932}, ls_id:{id:1}, ctx:0x14b055e36050, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, log_sync_used_time=152336, ctx_lock_wait_time=6) [2024-04-24 10:00:52.943490] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [862266][T1002_LSArchive][T1002][YB420A000003-000616C2AE58F0E7-0-0] [lt=93] gc stale ls task succ [2024-04-24 10:00:52.949457] INFO [LIB] stat (utility.h:1197) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=77] [APPLY STAT CB APPEND COST TIME](cur_stat_count=5, stat_interval=5000000, avg (count/cost)=411, this=0x14b133646280, extra_info="") [2024-04-24 10:00:52.949579] INFO [LIB] stat (utility.h:1197) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=102] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=5, stat_interval=5000000, avg (count/cost)=221539, this=0x14b1336464b0, extra_info="") [2024-04-24 10:00:52.949677] INFO [LIB] stat (utility.h:1197) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=75] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=5, stat_interval=5000000, avg (count/cost)=0, this=0x14b1336466e0, extra_info="") [2024-04-24 10:00:52.949813] INFO [LIB] stat (utility.h:1197) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=123] [APPLY STAT CB EXECUTE TIME](cur_stat_count=5, stat_interval=5000000, avg (count/cost)=-718, this=0x14b133646910, extra_info="") [2024-04-24 10:00:52.949928] INFO [LIB] stat (utility.h:1197) [862235][T1002_ApplySrv0][T1002][YB420A000003-000616C2BE49483E-0-0] [lt=101] [APPLY STAT CB TOTAL TIME](cur_stat_count=5, stat_interval=5000000, avg (count/cost)=221233, this=0x14b133646b40, extra_info="") [2024-04-24 10:00:52.954434] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=76] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.959643] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2272) [2326786][T1002_L0_G1][T1002][YB420A000002-000616C2C20C019D-0-0] [lt=93][errcode=-4389] log_task life cost too much time(palf_id=1002, self="10.0.0.3:2882", log_id=301045, log_task={header:{begin_lsn:{lsn:165109360}, end_lsn:{lsn:165110083}, log_id:301045, min_scn:{val:1713924052755549000, v:0}, max_scn:{val:1713924052755549000, v:0}, data_len:667, proposal_id:5, prev_lsn:{lsn:165107787}, prev_proposal_id:5, committed_end_lsn:{lsn:165109360}, data_checksum:2315775765, accum_checksum:3702960698, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713924052773499, freeze_ts:1713924052773519, submit_ts:1713924052773599, flushed_ts:1713924052783301, gen_to_freeze cost time:20, gen_to_submit cost time:100, submit_to_flush cost time:9702}, fs_cb_begin_ts=1713924052959593, log_life_time=186094) [2024-04-24 10:00:52.961842] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [862309][T1001_Occam][T1001][YB420A000003-000616C2AFF988F4-0-0] [lt=92] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:52.963612] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=93] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.963825] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [861893][T1_L0_G0][T1][YB420A000003-000616C2ADD8FCD9-0-0] [lt=99][errcode=0] set logical del time(cache_obj->get_logical_del_time()=576365537942, cache_obj->added_lc()=false, cache_obj->get_object_id()=39601, cache_obj->get_tenant_id()=1, lbt()="0x121f36ac 0xc327dd4 0x4e3fdbe 0x4e31e28 0x4e29b67 0x4e21201 0x4e1eb17 0x4e0e175 0xa7ac354 0x12d1fdbc 0x14b1c1d132de 0x14b1c1a44e83") [2024-04-24 10:00:52.965038] INFO eloop_run (eloop.c:143) [861404][pnio2][T0][Y0-0000000000000000-0-0] [lt=97] PNIO [ratelimit] time: 1713924052965030, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007589, add_bytes: 0 [2024-04-24 10:00:52.965232] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [862402][T1001_TenantMet][T1001][Y0-0000000000000000-0-0] [lt=142] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:769, free_obj_hold_cnt:74, allocator used:3351768, allocator total:3466624}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:72, free_obj_hold_cnt:32, allocator used:6821464, allocator total:6821464}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():154752, full allocator used:1600, full allocator total:220288}, tablets_mem=10508376, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:23, free_obj_hold_cnt:140, allocator used:323392, allocator total:392448}, pending_cnt=0, wait_gc_count=0, tablet count=841) [2024-04-24 10:00:52.972056] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=226][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:52.973206] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=191] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=7373) [2024-04-24 10:00:52.973313] INFO [LIB] stat (utility.h:1197) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=81] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x14b0efafce50, extra_info="id:1") [2024-04-24 10:00:52.973510] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2319281][T1004_L0_G0][T1004][YB420A000002-000616C2C34C0159-0-0] [lt=98] transaction rpc statistics(total_rt=36674, total_process=67, avg_rt=539) [2024-04-24 10:00:52.974044] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2136) [874624][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=89][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, lsn_array=[{lsn:298532778}], scn_array=[{val:1713924052929711000, v:0}], curr_size=122, accum_size=5450, time_cost=10167) [2024-04-24 10:00:52.975849] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [874542][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=111] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1002}, {id:1}]) [2024-04-24 10:00:52.977071] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [862253][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=96] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=1, total_destructed_detector=1, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x14b168fe42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-04-24 10:00:52.978782] INFO [LIB] stat (utility.h:1197) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=253] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=69834, this=0x14b0efb333b0, extra_info="palf_id:1") [2024-04-24 10:00:52.978937] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=146] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=18826) [2024-04-24 10:00:52.979038] INFO [LIB] stat (utility.h:1214) [874405][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=69] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=205, this=0x14b176ffc530, extra_info="") [2024-04-24 10:00:52.979116] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=111] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.0.0.3:2882", committed size=18524) [2024-04-24 10:00:52.979235] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=81] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=167, this=0x14b0efb335e0, extra_info="palf_id:1") [2024-04-24 10:00:52.984842] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=209] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:52.985243] INFO [LIB] stat (utility.h:1197) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=192] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=107, this=0x14b0c0e2b470, extra_info="") [2024-04-24 10:00:52.985349] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=73] [PALF STAT GROUP LOG INFO](palf_id=1001, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=6, avg_log_batch_cnt=1, total_group_log_size=5452, avg_group_log_size=908) [2024-04-24 10:00:52.985481] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=92] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:52.985575] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=84] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:52.991958] INFO [LIB] stat (utility.h:1197) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=169] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=72943, this=0x14b0c0e353b0, extra_info="palf_id:1001") [2024-04-24 10:00:52.992055] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2139) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=90] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, accum_size=6054) [2024-04-24 10:00:52.992158] INFO [LIB] stat (utility.h:1214) [874406][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=60] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=284, this=0x14b176ffcc70, extra_info="") [2024-04-24 10:00:52.992225] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=80] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="10.0.0.3:2882", committed size=7507) [2024-04-24 10:00:52.992338] INFO [LIB] stat (utility.h:1197) [874404][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=69] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=140, this=0x14b0c0e355e0, extra_info="palf_id:1001") [2024-04-24 10:00:52.999750] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2720) [861318][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4002] get invalid Ethernet speed, use default(devname="eth0") [2024-04-24 10:00:52.999848] WDIAG [SERVER] runTimerTask (ob_server.cpp:3239) [861318][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4002] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4002, ret="OB_INVALID_ARGUMENT") [2024-04-24 10:00:53.001615] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [862447][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=86] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:53.012147] INFO [LIB] stat (utility.h:1197) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=78] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=73, this=0x14b1470ab470, extra_info="") [2024-04-24 10:00:53.012261] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=90] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=6, avg_log_batch_cnt=1, total_group_log_size=1108, avg_group_log_size=184) [2024-04-24 10:00:53.012420] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=136] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=986) [2024-04-24 10:00:53.012512] INFO [LIB] stat (utility.h:1197) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=59] [PALF STAT APPEND COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=0, this=0x14b147074e50, extra_info="id:1") [2024-04-24 10:00:53.012641] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=121] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:53.012958] INFO [LIB] stat (utility.h:1197) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=116] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=105, this=0x14b1336eb470, extra_info="") [2024-04-24 10:00:53.013057] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=66] [PALF STAT GROUP LOG INFO](palf_id=1001, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=8, avg_log_batch_cnt=1, total_group_log_size=6779, avg_group_log_size=847) [2024-04-24 10:00:53.013201] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=128] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"10.0.0.3:2882", has_set_deleted:false}, append size=6779) [2024-04-24 10:00:53.013369] INFO [LIB] stat (utility.h:1197) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=77] [PALF STAT APPEND COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x14b134390e50, extra_info="id:1001") [2024-04-24 10:00:53.013491] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=112] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:53.013583] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=81] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:53.016073] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=79] current clock is (clock_=0, max_seq_=0, mem_limit=3181127926, cur_mem_hold=297795584, attr_.tenant_id_=1004) [2024-04-24 10:00:53.020505] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59CD-0-0] [lt=105] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=9, this=0x14b0c0ef7470, extra_info="") [2024-04-24 10:00:53.020674] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59CD-0-0] [lt=103] [PALF STAT GROUP LOG INFO](palf_id=1002, self="10.0.0.3:2882", role="FOLLOWER", total_group_log_cnt=5, avg_log_batch_cnt=0, total_group_log_size=937, avg_group_log_size=187) [2024-04-24 10:00:53.020776] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1748) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59CD-0-0] [lt=90] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="10.0.0.3:2882", committed size=1386) [2024-04-24 10:00:53.021136] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59CD-0-0] [lt=202] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=25, this=0x14b0c0ef76a0, extra_info="palf_id:1002") [2024-04-24 10:00:53.021204] INFO [PALF] read (log_cache.cpp:100) [874561][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=259] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1386, hit_cnt=5, read_cnt=5, hit rate=1.000000000000000000e+00) [2024-04-24 10:00:53.021377] INFO [LIB] stat (utility.h:1197) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59CD-0-0] [lt=218] [PALF STAT LOG LIFE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=240498, this=0x14b0c0ef78d0, extra_info="palf_id:1002") [2024-04-24 10:00:53.021519] INFO [PALF] sliding_cb (log_sliding_window.cpp:2261) [2325522][T1004_L0_G1][T1004][YB420A000002-000616C2BF8C59CD-0-0] [lt=124] [PALF STAT LOG TASK TIME](palf_id=1002, self="10.0.0.3:2882", total_slide_log_cnt=6, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=78, avg_log_submit_to_flush_time=179145, avg_log_submit_to_slide_time=240420) [2024-04-24 10:00:53.021775] INFO [LIB] stat (utility.h:1197) [874557][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=69] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=290, this=0x14b1312048f8, extra_info="") [2024-04-24 10:00:53.021962] INFO [LIB] stat (utility.h:1197) [874557][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=161] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=223, this=0x14b131204b28, extra_info="") [2024-04-24 10:00:53.024150] INFO [LIB] stat (utility.h:1197) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=104] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=110, this=0x14b0f0aa1470, extra_info="") [2024-04-24 10:00:53.024316] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=128] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=4641, avg_group_log_size=515) [2024-04-24 10:00:53.024414] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=75] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=4099) [2024-04-24 10:00:53.024501] INFO [LIB] stat (utility.h:1197) [874736][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=65] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x14b0f0a74e50, extra_info="id:1") [2024-04-24 10:00:53.025290] INFO [LIB] stat (utility.h:1197) [862260][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=168] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=107, this=0x14b14c9b3470, extra_info="") [2024-04-24 10:00:53.025401] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [874526][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=90] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1004, min_version={val:1713924052448472000, v:0}, base_version={val:1713924052086135000, v:0}, server_count=2, svr_array=[{addr:"10.0.0.2:2882", version:{val:1713924052130828000, v:0}, valid_part_count:0, total_part_count:3, generate_tstamp:1713924052968706, is_skipped:true}, {addr:"10.0.0.3:2882", version:{val:1713924052448472000, v:0}, valid_part_count:2, total_part_count:3, generate_tstamp:1713924053025264, is_skipped:false}], skip_server_count=1, skip_servers=["10.0.0.2:2882"]) [2024-04-24 10:00:53.025440] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [862260][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=111] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=4621, avg_group_log_size=513) [2024-04-24 10:00:53.027759] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:312) [862475][T1002_ArcFetche][T1002][YB420A000003-000616C2B158F0E7-0-0] [lt=69] ObArchiveFetcher is running(thread_index=1) [2024-04-24 10:00:53.032123] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [861686][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=100] get wrs ts(ls_id={id:1}, delta=109397, log_service_ts=1713924052921901, min_tx_service_ts=4611686018427387, timestamp={val:1713924052921901000, v:0}) [2024-04-24 10:00:53.032206] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [861686][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=77] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=599, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713924044692254001, v:0}", last_lsn={lsn:179729364}, last_gts={val:0, v:0}, min_start_scn="{val:1713924044653550000, v:0}", min_start_status=2, sys_ls_end_scn={val:0, v:0}) [2024-04-24 10:00:53.038014] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=134][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:53.038620] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=122] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:53.039492] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [874434][T1004_PlanCache][T1004][Y0-0000000000000000-0-0] [lt=67] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1004) [2024-04-24 10:00:53.039614] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [874434][T1004_PlanCache][T1004][Y0-0000000000000000-0-0] [lt=101] start lib cache evict(tenant_id=1004, mem_hold=457179136, mem_limit=579820585, cache_obj_num=2610, cache_node_num=2604) [2024-04-24 10:00:53.039787] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [874434][T1004_PlanCache][T1004][Y0-0000000000000000-0-0] [lt=161] end lib cache evict(tenant_id=1004, cache_evict_num=0, mem_hold=457179136, mem_limit=579820585, cache_obj_num=2610, cache_node_num=2604) [2024-04-24 10:00:53.039856] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [874434][T1004_PlanCache][T1004][Y0-0000000000000000-0-0] [lt=60] schedule next cache evict task(evict_interval=5000000) [2024-04-24 10:00:53.045545] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [874434][T1004_PlanCache][T1004][Y0-0000000000000000-0-0] [lt=113] Dumping All Cache Objs(alloc_obj_list.count()=2610, alloc_obj_list=[{obj_id:270, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173600}, {obj_id:271, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174960}, {obj_id:280, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:281, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:282, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:283, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:284, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:286, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:288, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:289, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:291, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:292, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:295, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:296, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:299, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:300, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178499}, {obj_id:301, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:302, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:303, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:304, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:307, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:309, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:310, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:311, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:313, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:314, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:315, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:316, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:317, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:318, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:319, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:320, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:321, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:322, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:323, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:324, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:325, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:326, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:327, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:328, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:329, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:330, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:331, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:332, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:333, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:334, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:335, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:336, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:337, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:338, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:339, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:340, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:341, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:342, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:343, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:344, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:345, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:346, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:347, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:348, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:349, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:350, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:351, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:352, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:353, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:354, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:355, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:356, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:357, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:358, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:359, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:360, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:361, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:362, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:363, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:364, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:365, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:366, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:367, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:368, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:369, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:370, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:371, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:372, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:373, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:374, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:375, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:376, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:381, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:382, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:383, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:384, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:385, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:386, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:387, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:388, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:389, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:390, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:391, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:392, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:393, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:394, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:395, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:396, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:397, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:398, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:399, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:400, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:401, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:402, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:403, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:404, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:405, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:406, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:407, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:408, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:409, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:410, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:411, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:412, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:413, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:415, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173296}, {obj_id:419, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:301071}, {obj_id:420, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246208}, {obj_id:421, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281861}, {obj_id:422, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268960}, {obj_id:423, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214161}, {obj_id:424, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253263}, {obj_id:425, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175408}, {obj_id:426, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178442}, {obj_id:427, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228175}, {obj_id:428, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186751}, {obj_id:429, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195314}, {obj_id:430, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173248}, {obj_id:431, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299295}, {obj_id:432, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232430}, {obj_id:433, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203884}, {obj_id:434, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210014}, {obj_id:435, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222957}, {obj_id:436, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:205775}, {obj_id:437, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:218381}, {obj_id:438, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197447}, {obj_id:439, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197445}, {obj_id:440, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191492}, {obj_id:441, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212278}, {obj_id:442, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197938}, {obj_id:445, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198158}, {obj_id:446, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188509}, {obj_id:447, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185401}, {obj_id:448, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184116}, {obj_id:449, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198513}, {obj_id:450, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191573}, {obj_id:451, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186879}, {obj_id:452, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184292}, {obj_id:453, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187220}, {obj_id:454, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184324}, {obj_id:455, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187221}, {obj_id:456, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184260}, {obj_id:457, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190471}, {obj_id:459, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185777}, {obj_id:460, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182076}, {obj_id:461, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208545}, {obj_id:462, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181884}, {obj_id:463, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186650}, {obj_id:464, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182284}, {obj_id:465, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190347}, {obj_id:466, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181980}, {obj_id:467, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183132}, {obj_id:468, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187473}, {obj_id:469, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175168}, {obj_id:470, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177704}, {obj_id:471, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202250}, {obj_id:472, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:305642}, {obj_id:473, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180374}, {obj_id:474, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:249607}, {obj_id:475, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184924}, {obj_id:476, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172304}, {obj_id:477, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:254861}, {obj_id:478, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:301987}, {obj_id:479, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245954}, {obj_id:482, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187993}, {obj_id:484, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226929}, {obj_id:485, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241889}, {obj_id:486, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220521}, {obj_id:487, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235257}, {obj_id:489, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185912}, {obj_id:490, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185912}, {obj_id:492, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182824}, {obj_id:493, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193849}, {obj_id:494, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187348}, {obj_id:495, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187372}, {obj_id:496, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:497, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173504}, {obj_id:498, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:499, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177273}, {obj_id:500, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:502, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181132}, {obj_id:503, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:504, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216328}, {obj_id:505, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179603}, {obj_id:506, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:507, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:509, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178892}, {obj_id:510, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174688}, {obj_id:512, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180165}, {obj_id:513, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186392}, {obj_id:514, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182904}, {obj_id:515, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180509}, {obj_id:516, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179088}, {obj_id:517, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180853}, {obj_id:518, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179119}, {obj_id:519, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180854}, {obj_id:520, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179062}, {obj_id:521, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182000}, {obj_id:522, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182404}, {obj_id:523, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179995}, {obj_id:524, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177793}, {obj_id:525, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191818}, {obj_id:526, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177809}, {obj_id:527, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180277}, {obj_id:528, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178217}, {obj_id:529, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182256}, {obj_id:530, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177905}, {obj_id:531, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178489}, {obj_id:532, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181655}, {obj_id:533, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:286440}, {obj_id:534, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:255974}, {obj_id:535, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:280815}, {obj_id:536, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:250354}, {obj_id:537, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180836}, {obj_id:538, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:539, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183476}, {obj_id:541, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187928}, {obj_id:542, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:543, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177119}, {obj_id:544, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:545, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179069}, {obj_id:546, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198078}, {obj_id:547, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172528}, {obj_id:548, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175256}, {obj_id:550, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217216}, {obj_id:551, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178297}, {obj_id:553, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182203}, {obj_id:555, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186953}, {obj_id:556, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180347}, {obj_id:558, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:559, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172080}, {obj_id:560, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191778}, {obj_id:561, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:562, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179119}, {obj_id:563, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187733}, {obj_id:564, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186020}, {obj_id:565, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:568, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178398}, {obj_id:569, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185485}, {obj_id:570, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:571, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194633}, {obj_id:572, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194633}, {obj_id:573, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183528}, {obj_id:574, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183528}, {obj_id:575, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172960}, {obj_id:576, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173696}, {obj_id:578, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:580, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174080}, {obj_id:583, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203274}, {obj_id:584, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180108}, {obj_id:585, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215880}, {obj_id:586, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:588, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203466}, {obj_id:590, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228543}, {obj_id:592, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:171872}, {obj_id:593, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175032}, {obj_id:594, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228351}, {obj_id:595, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184284}, {obj_id:596, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:598, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:599, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:600, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176736}, {obj_id:601, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:602, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175416}, {obj_id:603, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173104}, {obj_id:604, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:605, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210897}, {obj_id:606, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178793}, {obj_id:607, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:609, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289757}, {obj_id:612, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176820}, {obj_id:613, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:617, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178414}, {obj_id:618, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189901}, {obj_id:619, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272691}, {obj_id:620, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216666}, {obj_id:621, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:623, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181615}, {obj_id:624, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187140}, {obj_id:625, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:628, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:205860}, {obj_id:630, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198783}, {obj_id:631, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179440}, {obj_id:632, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212784}, {obj_id:633, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202696}, {obj_id:634, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199982}, {obj_id:635, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:636, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:639, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178921}, {obj_id:640, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185967}, {obj_id:641, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179568}, {obj_id:642, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:643, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:644, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:645, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:646, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:647, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:648, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:649, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:650, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:288992}, {obj_id:655, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160946}, {obj_id:658, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176907}, {obj_id:659, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176232}, {obj_id:660, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172912}, {obj_id:661, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185552}, {obj_id:662, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202696}, {obj_id:663, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199982}, {obj_id:664, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178921}, {obj_id:665, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185967}, {obj_id:666, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179568}, {obj_id:667, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:668, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:669, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:166013}, {obj_id:670, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:162314}, {obj_id:671, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157852}, {obj_id:672, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:673, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175542}, {obj_id:674, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157671}, {obj_id:677, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:678, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:679, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208603}, {obj_id:680, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186331}, {obj_id:681, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201118}, {obj_id:682, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186330}, {obj_id:683, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192344}, {obj_id:684, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177824}, {obj_id:685, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:686, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:687, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:688, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:689, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:690, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:691, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:692, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:693, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:694, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:695, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:696, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:697, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:698, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:699, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:700, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:702, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:703, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:705, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453841}, {obj_id:706, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174520}, {obj_id:708, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:709, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:710, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:711, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:712, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:713, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:714, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:715, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:716, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:717, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:718, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:719, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:720, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:721, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:722, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:723, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:724, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:725, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:726, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:727, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:728, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:729, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:730, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:731, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:732, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:733, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:734, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180314}, {obj_id:735, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182920}, {obj_id:736, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222053}, {obj_id:737, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:382675}, {obj_id:738, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203482}, {obj_id:739, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:740, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:741, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:742, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:743, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:744, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:745, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:746, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:747, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:748, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:749, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:750, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:751, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:752, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:753, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:754, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:755, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:756, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:757, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:758, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:759, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:760, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:761, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:762, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:763, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:764, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:765, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:766, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:767, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:768, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:769, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:770, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:771, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:772, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:773, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:774, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:775, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:776, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:777, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:778, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:779, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:780, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:781, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:782, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:783, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:784, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:785, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:786, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:787, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:788, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:789, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:790, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:791, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:792, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:793, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count [2024-04-24 10:00:53.047033] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [874434][T1004_PlanCache][T1004][Y0-0000000000000000-0-0] [lt=1470] schedule next cache evict task(evict_interval=5000000) [2024-04-24 10:00:53.047965] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=67] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-04-24 10:00:53.049615] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [861771][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=74] current monitor number(seq_=-1) [2024-04-24 10:00:53.053168] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:213) [862418][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=86] start to check and clear obj lock(ls->get_ls_id()={id:1}) [2024-04-24 10:00:53.054797] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=83] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:53.063484] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=98] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:53.072316] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=65] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"10.0.0.3:2882", switch_to_leader_ts:-1}) [2024-04-24 10:00:53.072453] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=104] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"10.0.0.3:2882", switch_to_leader_ts:-1}) [2024-04-24 10:00:53.072554] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=95][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:53.080206] INFO [STORAGE] exec_wait (ob_tmp_file_cache.cpp:1531) [1556243][T1004_ComTimerT][T1004][Y0-0000000000000000-0-0] [lt=99] succeed to do one round of tmp block io(ret=-4023, loop_nums=0, wait_io_cnt=0, washing_count=0, block_cache_num=0, page_cache_num=0) [2024-04-24 10:00:53.091227] INFO [LIB] stat (utility.h:1197) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=183] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=102, this=0x14b0efb29470, extra_info="") [2024-04-24 10:00:53.091373] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1091) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=103] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.0.0.3:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=7791, avg_group_log_size=779) [2024-04-24 10:00:53.091537] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=127] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:53.091609] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=58] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:53.091792] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=174] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:53.096382] INFO eloop_run (eloop.c:143) [861399][pnio1][T0][Y0-0000000000000000-0-0] [lt=111] PNIO [ratelimit] time: 1713924053096350, bytes: 14677211866, bw: 0.287247 MB/s, add_ts: 1003824, add_bytes: 302352 [2024-04-24 10:00:53.098777] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [862344][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=68] ====== tenant freeze timer task ====== [2024-04-24 10:00:53.098947] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [862344][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=136] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=96468992, ctx.total_memstore_used_=130023424, ctx.total_memstore_hold_=130023424, memstore_freeze_trigger_limit_=107374180, tenant_id=1001) [2024-04-24 10:00:53.099025] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [862344][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=71] [TenantFreezer] oldest frozen memtable(list="") [2024-04-24 10:00:53.112136] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=66] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:53.112206] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=57] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:53.112270] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=58] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:53.126319] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:270) [862309][T1001_Occam][T1001][YB420A000003-000616C2AFF988F6-0-0] [lt=188][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::process_with_tablet_(oceanbase::storage::ObTablet &), (*this=|threshold=10.00ms|start at 10:00:53.115|414=8us|416=33us|418=72us|total=10.69ms) [2024-04-24 10:00:53.130185] INFO [SQL.JO] compute_table_location (ob_join_order.cpp:278) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=70] succeed to calculate base table sharding info(table_id=18446744073709551614, ref_table_id=343, is_global_index=false) [2024-04-24 10:00:53.130285] INFO [SQL.JO] init_est_sel_info_for_access_path (ob_join_order.cpp:12189) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=63] total rowcount, use statistics(table_meta_info_.table_row_count_=654, table_meta_info_.average_row_size_=1.200000000000000000e+02, table_meta_info_.micro_block_count_=0, table_meta_info_.part_size_=7.848000000000000000e+04) [2024-04-24 10:00:53.132224] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [861686][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=76] get_max_applied_scn(scn={val:1, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1, v:0}}) [2024-04-24 10:00:53.132314] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [861686][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=82] get_min_unreplayed_log_info(lsn={lsn:179759201}, scn={val:1713924052921901001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713924053023841, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:18156, ret_code_:0}}, next_to_submit_lsn_:{lsn:179759201}, next_to_submit_scn_:{val:1713924052921901001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x14b11a8c9000, next_round_pread_size:3678208, curr_read_pos:1005, curr_read_buf_start_pos:0, curr_read_buf_end_pos:1005, log_storage_:{IteratorStorage:{start_lsn:{lsn:179758196}, end_lsn:{lsn:179759201}, read_buf:{buf_len_:389120, buf_:0x14b11a8c9000}, block_size:67104768, log_storage_:0x14b186b4d470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713924052921901000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:917, scn_:{val:1713924052921901000, v:0}, data_checksum:140950650, flag:1}}, init_mode_version:0, accumulate_checksum:1678844235, curr_entry_is_padding:0, padding_entry_size:949, padding_entry_scn:{val:1713924052921901000, v:0}}}}}) [2024-04-24 10:00:53.135728] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [861387][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=185] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-8907464704, global_cache_size=630472704, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1, 1001, 1002, 1003, 1004]) [2024-04-24 10:00:53.136313] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=68] current clock is (clock_=0, max_seq_=0, mem_limit=3181127926, cur_mem_hold=297795584, attr_.tenant_id_=1004) [2024-04-24 10:00:53.138184] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=84][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:53.138255] INFO [SHARE] deserialize_ (ob_rpc_struct.cpp:9227) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=88] deserialize txDesc from session(tx_desc={this:0x14b17299e780, tx_id:{txid:176927}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924052086137, op_sn:23, alloc_ts:-1, active_ts:1713924052085377, commit_ts:-1, finish_ts:-1, timeout_us:999997425, lock_timeout_us:-1, expire_ts:1713925052082802, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430293789517725, first_scn:1713924052095499, last_scn:1713924053135391, last_touch_ts:0}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, this={tenant_id:1004, tx_desc:{this:0x14b17299e780, tx_id:{txid:176927}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924052086137, op_sn:23, alloc_ts:-1, active_ts:1713924052085377, commit_ts:-1, finish_ts:-1, timeout_us:999997425, lock_timeout_us:-1, expire_ts:1713925052082802, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430293789517725, first_scn:1713924052095499, last_scn:1713924053135391, last_touch_ts:0}], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, ls_id:{id:1001}, type:3, buf_.length():0x9f, request_id:1, register_flag:{need_flush_redo_instantly_:false, mds_base_scn_:{val:18446744073709551615, v:3}}}) [2024-04-24 10:00:53.138834] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=73] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:53.138852] INFO [STORAGE] create_tablet (ob_ls_tablet_service.cpp:1972) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=111] no need to create sstable(ls_id={id:1001}, tablet_id={id:1152921504606847583}, create_tablet_schema={ObStorageSchema:{this:0x14b11ebc6060, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:0, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1713924052938480, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[{meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:1, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:1, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}]}, table_id:500929, index_status:1, truncate_version:-1}) [2024-04-24 10:00:53.139013] INFO [STORAGE] init (ob_medium_compaction_mgr.cpp:155) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=151] success to init medium clog recorder(ret=0, ls_id={id:1001}, tablet_id={id:1152921504606847583}, max_saved_version=0) [2024-04-24 10:00:53.139147] INFO [STORAGE.TRANS] init_storage_recorder (ob_tablet_memtable_mgr.cpp:149) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=87] success to init storage recorder(ret=0, ls_id={id:1001}, tablet_id={id:1152921504606847583}, max_saved_schema_version=1713924052938480, max_saved_medium_scn=0, compat_mode=0) [2024-04-24 10:00:53.139309] INFO [STORAGE] init_for_first_time_creation (ob_tablet.cpp:263) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=71] succeeded to init tablet for first time creation(ret=0, sstable=NULL, *this={this:0x14b04a6e5fa0, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:2, length:0, tablet_addr:[0-0-0-3458765810900664320-0](offset=0,size=1208,type=3,seq=0), allocator:0x14b0e25e2b80, tablet_meta:{version:1, ls_id:{id:1001}, tablet_id:{id:1152921504606847583}, data_tablet_id:{id:200317}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1713895200302650000, multi_version_start:1713895200302650000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1713895200302650000, cur_report_version:1713895200302650000, data_checksum:0, row_count:0}, table_store_flag:{with_major_sstable:0}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1713924052938480, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1713924052938480}, table_store_addr:{ptr:{this:0x14b04a6e6460, version:257, major_tables:{cnt:0, is_inited:false}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:0}, is_ready_for_read:false, sstable_arrays:{major_tables:{null}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{null}}}, addr:[0-0-0-3458764857417924608-0](offset=0,size=320,type=3,seq=0)}, storage_schema_addr:{ptr:{this:0x14b04a6e65b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:0, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1713924052938480, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[{meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:1, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}, {meta_type:{type:"INT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, is_column_stored_in_sstable:1, is_rowkey_column:1, is_generated_column:0, orig_default_value:{"NULL":"NULL"}}]}, addr:[0-0-0-3458764814468251648-0](offset=0,size=280,type=3,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:0}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:7, str:"MAX"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:0, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0}) [2024-04-24 10:00:53.139545] INFO [STORAGE.TRANS] create_tablet (ob_tablet_to_ls_cache.cpp:135) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=182] create tablet cache(ret=0, ret="OB_SUCCESS", tablet_id={id:1152921504606847583}, ls_id={id:1001}) [2024-04-24 10:00:53.139955] INFO [STORAGE] get_tablet_report_info (ob_tablet.cpp:4405) [861423][TbltTblUp0][T1004][YB420A000003-000616C2AB2901F6-0-0] [lt=60] no major sstables in this tablet, cannot report(ret=-4725) [2024-04-24 10:00:53.139977] INFO [MDS] register_to_mds_table_mgr (mds_table_mgr.cpp:100) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=88] register success(ret=0, ret="OB_SUCCESS", p_mds_table={this:0x14b0d6aadc20, ls_id:{id:1001}, tablet_id:{id:1152921504606847583}, flushing_scn:{val:18446744073709551615, v:3}, rec_scn:{val:4611686018427387903, v:0}, last_inner_recycled_scn:{val:0, v:0}, total_node_cnt:0, construct_sequence:12263, debug_info:{do_init_tablet_pointer:0x14b051fd5278, do_remove_tablet_pointer:null, init_ts:"2024-04-24 10:00:53.139948", last_reset_ts:"1970-01-01 08:00:00.0", remove_ts:"1970-01-01 08:00:00.0", last_flush_ts:"1970-01-01 08:00:00.0", switch_to_empty_shell_ts:"1970-01-01 08:00:00.0", init_trace_id:YB420A000002-000616C2C30C01F3-0-0, remove_trace_id:Y0-0000000000000000-0-0}}) [2024-04-24 10:00:53.140033] WDIAG [SERVER] inner_fill_tablet_info_ (ob_service.cpp:2451) [861423][TbltTblUp0][T1004][YB420A000003-000616C2AB2901F6-0-0] [lt=71][errcode=-4725] fail to get tablet report info from tablet(ret=-4725, ret="OB_TABLET_NOT_EXIST", tenant_id=1004, tablet_id={id:1152921504606847583}) [2024-04-24 10:00:53.140076] INFO [MDS] init (mds_table_base.cpp:76) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=77] mds table inited(ret=0, ret="OB_SUCCESS", *this={this:0x14b0d6aadc20, ls_id:{id:1001}, tablet_id:{id:1152921504606847583}, flushing_scn:{val:18446744073709551615, v:3}, rec_scn:{val:4611686018427387903, v:0}, last_inner_recycled_scn:{val:0, v:0}, total_node_cnt:0, construct_sequence:12263, debug_info:{do_init_tablet_pointer:0x14b051fd5278, do_remove_tablet_pointer:null, init_ts:"2024-04-24 10:00:53.139948", last_reset_ts:"1970-01-01 08:00:00.0", remove_ts:"1970-01-01 08:00:00.0", last_flush_ts:"1970-01-01 08:00:00.0", switch_to_empty_shell_ts:"1970-01-01 08:00:00.0", init_trace_id:YB420A000002-000616C2C30C01F3-0-0, remove_trace_id:Y0-0000000000000000-0-0}}) [2024-04-24 10:00:53.140182] WDIAG [SERVER] do_batch_update_ (ob_tablet_table_updater.cpp:613) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F6-0-0] [lt=64][errcode=-4002] tasks num not match or invalid tablet_operator(ret=-4002, ret="OB_INVALID_ARGUMENT", task_cnt=0, replica_cnt=0, checksum_cnt=0) [2024-04-24 10:00:53.140249] INFO [SERVER] do_batch_update_ (ob_tablet_table_updater.cpp:645) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F6-0-0] [lt=54] REPORT: batch update tablets finished(ret=-4002, ret="OB_INVALID_ARGUMENT", replicas.count()=0, cost_time=70) [2024-04-24 10:00:53.140327] WDIAG [SERVER] batch_process_tasks (ob_tablet_table_updater.cpp:514) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F6-0-0] [lt=71][errcode=-4002] do_batch_update_ failed(tmp_ret=-4002, tmp_ret="OB_INVALID_ARGUMENT", start_time=1713924053139919, tasks count=0, tablet replicas count=0) [2024-04-24 10:00:53.140379] INFO [MDS.EVENT]construct_insert_record_user_mds_node_ (mds_row.ipp:216) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=119] WRITE_NODE(key={tenant_id:1004, ls_id:{id:1001}, tablet_id:{id:1152921504606847583}}, event={alloc:null, timestamp:"2024-04-24 10:00:53.140360", event:"WRITE_NODE", info_str:"{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}", unit_id:0, key_str:"Dummy", writer_type:1, writer_id:176927, seq_no:0, redo_scn:{val:4611686018427387903, v:0}, end_scn:{val:4611686018427387903, v:0}, trans_version:{val:4611686018427387903, v:0}, node_type:1, state:0}) [2024-04-24 10:00:53.140563] INFO [STORAGE] set_tablet_status (ob_ls_tablet_service.cpp:2382) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=92] succeeded to set tablet status(ret=0, tablet_id={id:1152921504606847583}, tablet_status={tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:18446744073709551615, v:3}, create_commit_version:-1, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}) [2024-04-24 10:00:53.140775] INFO [STORAGE] set_tablet_gc_trigger (ob_tablet_gc_service.cpp:277) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=114] set_tablet_gc_trigger(this={tablet_persist_trigger:2, is_inited:true}) [2024-04-24 10:00:53.140851] INFO [MDS] register_process (ob_tablet_create_mds_helper.cpp:84) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=65] create tablet register(ret=0, ret="OB_SUCCESS", arg={ls_id:{id:1001}, major_frozen_scn:{val:1713895200302650000, v:0}, total_tablet_cnt:1, tablets:{ [874592] [YB420A000002-000616C2C30C01F3-0-0] data_tablet_id:{id:200317}, tablet_ids:[{id:1152921504606847583}], compat_mode:0, is_create_bind_hidden_tablets:false }}) [2024-04-24 10:00:53.141010] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=128] notify MDS(ret=0, trans_id_={txid:176927}, ls_id_={id:1001}, notify_type=0, log_ts={val:18446744073709551615, v:3}, notify_array.count()=1, notify_array=[{register_no_:0, has_submitted_:false, has_synced_:false, type:3, data_.length():159}], total_time=0) [2024-04-24 10:00:53.141096] INFO [STORAGE.TRANS] register_mds_into_tx (ob_trans_service.cpp:912) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=73] register multi data source result(ret=0, ret="OB_SUCCESS", arg={tenant_id:1004, tx_desc:{this:0x14b17299e780, tx_id:{txid:176927}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924052086137, op_sn:23, alloc_ts:-1, active_ts:1713924052085377, commit_ts:-1, finish_ts:-1, timeout_us:999997425, lock_timeout_us:-1, expire_ts:1713925052082802, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430293789517725, first_scn:1713924052095499, last_scn:1713924053135391, last_touch_ts:0}, {id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993244784220907, first_scn:1713924053136526, last_scn:1713924053136526, last_touch_ts:1713924053136528}], exec_info_reap_ts:1713924053136527, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, ls_id:{id:1001}, type:3, buf_.length():0x9f, request_id:1, register_flag:{need_flush_redo_instantly_:false, mds_base_scn_:{val:18446744073709551615, v:3}}}, result={result:0, tx_result:{incomplete:false, parts:[{id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993244784220907, first_scn:1713924053136526, last_scn:1713924053136526, last_touch_ts:1713924053136527}], touched_ls_list:[], cflict_txs:[]}}, tx_desc={this:0x14b17299e780, tx_id:{txid:176927}, state:2, addr:"10.0.0.2:2882", tenant_id:1004, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:1713924052086137, op_sn:23, alloc_ts:-1, active_ts:1713924052085377, commit_ts:-1, finish_ts:-1, timeout_us:999997425, lock_timeout_us:-1, expire_ts:1713925052082802, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.0.0.3:2882", epoch:1430293789517725, first_scn:1713924052095499, last_scn:1713924053135391, last_touch_ts:0}, {id:{id:1001}, addr:"10.0.0.3:2882", epoch:1993244784220907, first_scn:1713924053136526, last_scn:1713924053136526, last_touch_ts:1713924053136528}], exec_info_reap_ts:1713924053136527, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1713874979, cluster_version:17180000768, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, local_retry_cnt=0, retry_cnt=0, request_id=1, time_guard=time guard have no click events for optimization) [2024-04-24 10:00:53.142706] INFO [SHARE] inner_batch_remove_by_sql_ (ob_tablet_replica_checksum_operator.cpp:442) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F6-0-0] [lt=77] will batch delete tablet replica checksum(affected_rows=0) [2024-04-24 10:00:53.142827] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [874742][T1003_LSArchive][T1003][YB420A000003-000616C2B608F0E7-0-0] [lt=126] gc stale ls task succ [2024-04-24 10:00:53.143349] INFO [SERVER] do_batch_remove_ (ob_tablet_table_updater.cpp:593) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F6-0-0] [lt=99] REPORT: batch remove tablets finished(ret=0, ret="OB_SUCCESS", tasks_count=1, cost_time=2932) [2024-04-24 10:00:53.143457] INFO [SERVER] dump (ob_tablet_table_updater.cpp:66) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F6-0-0] [lt=94] [TABLET_TABLE_UPDATE_STATISTIC] dump tablet table update statistics(suc_cnt=4, fail_cnt=0, remove_task_cnt=2, update_task_cnt=2, avg_wait_us=56853, avg_exec_us=82006) [2024-04-24 10:00:53.143585] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F6-0-0] [lt=122][errcode=-4002] fail to batch process task(ret=-4002) [2024-04-24 10:00:53.144250] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [861423][TbltTblUp0][T0][YB420A000003-000616C2AB2901F6-0-0] [lt=652][errcode=-4002] fail to batch execute task(ret=-4002, tasks.count()=1) [2024-04-24 10:00:53.154993] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=158] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:53.163649] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874755][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=67] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:53.169366] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:775) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04BB-0-0] [lt=99][errcode=-4023] tx is 2pc logging(this={this:0x14b148003290, ref:3, trans_id:{txid:176933}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924082953417, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052972925], ctx_create_time:1713924052965758}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052964375, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052971762000, v:0}, log_ts:{val:1713924052971762000, v:0}, lsn:{lsn:79646007}, submit_ts:1713924052971762}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176933}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176933}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924052971762000, v:0}, start_scn:{val:1713924052971762000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465782000 trans_version={val:1713924052971762000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176933} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=871402192 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052971762000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052972990, block_frozen_memtable:null}) [2024-04-24 10:00:53.169572] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:853) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04BB-0-0] [lt=199][errcode=-4023] trx commit failed(ret=-4023, ret="OB_EAGAIN", this={this:0x14b148003290, ref:3, trans_id:{txid:176933}, tenant_id:1004, is_exiting:false, trans_expired_time:1713924082953417, cluster_version:17180000768, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1713924052972925], ctx_create_time:1713924052965758}{ls_id:{id:1}, session_id:1, part_trans_action:3, pending_write:0, exec_info:{state:10, upstream:{id:-1}, participants:[{id:1}], incremental_participants:[], prev_record_lsn:{lsn:18446744073709551615}, redo_lsns:[], redo_log_no:0, multi_data_source:[], scheduler:"10.0.0.2:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_applying_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:1713924052964375, checksum:0, checksum_scn:{val:0, v:0}, max_durable_lsn:{lsn:18446744073709551615}, data_complete:false, is_dup_tx:false, prepare_log_info_arr:[], xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:", g_hv:0, b_hv:0}, need_checksum:true, is_sub2pc:false}, sub_state:{flag:26}, is_leaf():false, is_root():false, busy_cbs_.get_size():1, final_log_cb_:{ObTxBaseLogCb:{base_ts:{val:1713924052971762000, v:0}, log_ts:{val:1713924052971762000, v:0}, lsn:{lsn:79646007}, submit_ts:1713924052971762}, this:0x14b148006200, is_inited_:true, trans_id:{txid:176933}, ls_id:{id:1}, ctx:0x14b148003290, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}}, ctx_tx_data_:{ctx_mgr_:0x14b0db804030, tx_data_guard_:{tx_data:{tx_id:{txid:176933}, ref_cnt:1, state:"RUNNING", commit_version:{val:1713924052971762000, v:0}, start_scn:{val:1713924052971762000, v:0}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, read_only_:false}, role_state_:0, start_replay_ts_:{val:18446744073709551615, v:3}, start_recover_ts_:{val:18446744073709551615, v:3}, is_incomplete_replay_ctx_:false, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1713877465782000 max_table_version=1713877465782000 trans_version={val:1713924052971762000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:176933} ls_id=1 callback_alloc_count=1 callback_free_count=0 checksum=871402192 tmp_checksum=0 checksum_scn={val:4611686018427387903, v:0} redo_filled_count=2 redo_sync_succ_count=0 redo_sync_fail_count=0 main_list_length=2 unsynced_cnt=1 unsubmitted_cnt_=0 cb_statistics:[main=2, slave=0, merge=0, tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log_cb=0]}, coord_prepare_info_arr_:[], upstream_state:10, retain_cause:-1, 2pc_role:-1, collected:[], rec_log_ts:{val:1713924052971762000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, state_info_array:[], last_request_ts:1713924052972990, block_frozen_memtable:null}) [2024-04-24 10:00:53.169715] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1974) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04BB-0-0] [lt=136][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:176933}) [2024-04-24 10:00:53.169789] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1907) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04BB-0-0] [lt=65][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176933}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924053164760, cluster_id:1713874979}, expire_ts:1713924082968212, commit_start_scn:{val:1713924052958341000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:53.169964] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:141) [2322437][T1004_L0_G0][T1004][YB420A000002-000616C2BB2C04BB-0-0] [lt=129][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180000768, tenant_id:1004, tx_id:{txid:176933}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"10.0.0.2:2882", epoch:-1, request_id:4, timestamp:1713924053164760, cluster_id:1713874979}, expire_ts:1713924082968212, commit_start_scn:{val:1713924052958341000, v:0}, parts:[{id:1}], app_trace_info:""}) [2024-04-24 10:00:53.172518] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [862258][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=85][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:53.175557] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=133] notify MDS(ret=0, trans_id_={txid:176927}, ls_id_={id:1}, notify_type=0, log_ts={val:18446744073709551615, v:3}, notify_array.count()=1, notify_array=[{register_no_:0, has_submitted_:false, has_synced_:false, type:1, data_.length():76}], total_time=0) [2024-04-24 10:00:53.181388] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [862309][T1001_Occam][T1001][YB420A000003-000616C2AFF988F6-0-0] [lt=119] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-24 10:00:53.186479] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=90] notify MDS(ret=0, trans_id_={txid:176927}, ls_id_={id:1}, notify_type=0, log_ts={val:18446744073709551615, v:3}, notify_array.count()=1, notify_array=[{register_no_:0, has_submitted_:false, has_synced_:false, type:1, data_.length():76}], total_time=0) [2024-04-24 10:00:53.191249] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=148] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:53.191373] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=109] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:53.191431] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [874543][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=52] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:53.197929] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=87] notify MDS(ret=0, trans_id_={txid:176927}, ls_id_={id:1}, notify_type=0, log_ts={val:18446744073709551615, v:3}, notify_array.count()=1, notify_array=[{register_no_:0, has_submitted_:false, has_synced_:false, type:1, data_.length():67}], total_time=0) [2024-04-24 10:00:53.212306] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=68] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:53.212408] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=88] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2024-04-24 10:00:53.212474] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862277][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=60] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1002}) [2024-04-24 10:00:53.221117] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861498][RpcIO][T0][Y0-0000000000000000-0-0] [lt=63] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:53.221946] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861496][RpcIO][T0][Y0-0000000000000000-0-0] [lt=116] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:53.222005] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861497][RpcIO][T0][Y0-0000000000000000-0-0] [lt=60] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:53.222611] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [874754][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=87] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-24 10:00:53.227972] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [861495][RpcIO][T0][Y0-0000000000000000-0-0] [lt=106] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-24 10:00:53.229149] INFO [STORAGE.TRANS] notify_data_source_ (ob_trans_part_ctx.cpp:6591) [874592][T1004_L5_G0][T1004][YB420A000002-000616C2C30C01F3-0-0] [lt=158] notify MDS(ret=0, trans_id_={txid:176927}, ls_id_={id:1001}, notify_type=0, log_ts={val:18446744073709551615, v:3}, notify_array.count()=1, notify_array=[{register_no_:0, has_submitted_:false, has_synced_:false, type:1, data_.length():67}], total_time=0) [2024-04-24 10:00:53.238396] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [874524][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=71][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2024-04-24 10:00:53.239020] INFO [PALF] submit_log (palf_handle_impl.cpp:456) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=115] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.0.0.3:2882", has_set_deleted:false}, append size=3997) [2024-04-24 10:00:53.239163] INFO [LIB] stat (utility.h:1197) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=113] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x14b14c97ee50, extra_info="id:1") [2024-04-24 10:00:53.239258] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [862448][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=88] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-04-24 10:00:53.248329] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [861388][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=93] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-04-24 10:00:53.255167] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [861703][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=124] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1})